Notes from the Future
here are a few random thoughts about the future of tech.
Next up in sever farms, arm-mobile cores with system on chip gpus. I have direct word that google and facebook are both experimenting with this tech. nVidia is well positioned to take this market in 2-5 years with tegra2/3 / tegraX, and compete with intel/ amd in the low power server space.
There is a growing walled garden api problem. this results in the need for openid and oauth to support 4th party authentication, or granting someone the ability to delegate authority for you. you could think of this like if api's could act via proxy, representative, assistant, or agent. 4th party delegated identity will be huge, and will allow you to pay for agents that filter and aggregate socially restrictive data. google doesn't own this, and neither will facebook. there is room for a newcomer.
gov2.0 will happen, i am calling it #agilegov, and it will be less like democracy, and more like wikiocracy/ do-ocracies. others are calling it #bigsocial
3d games, will move into the web, and the web will finally move to 3d in the next decade. there are way to many gaps in platform tools for 3d worlds.
edu2.0, better connect top educators to more students. create smaller more engaging learning tools, or microlessons, that track behavior to a learning group online. think github for edu.
anyways that's my rant.
Rate of innovation, too high? Displacing our humanity?
If technology is moving so fast that the past and present are not predictors of future trends, that is a serious problem.
Continuity starts to fade.
Have you ever wondered why people put so much effort into saving a historic building or preserving some artifact? When I was a young technologist I would have razed it all if it meant progress could move faster, but today I am a bit wiser. In the last few years, I have been moving a lot, and have been disrupting my own personal continuity. New friends, new buildings, new places to eat, new everything. Even my family seems.... more distant, not because of distances, but in May my parents started pursuing a divorce. So, for me, nothing is the same, except a few core friends; some of which moved out here. My happiness went way up when they got here, and a small savory piece of continuity was restored for me.
So, what I have realized is that Continuity is what makes us human. And as a result we have rich pasts and experiences to draw from. We have all of recorded history to define and differentiate us. ( of only about 10,000 years ) What will humans know of us 10,000 years from now? What will be preserved? What will be our Iliad? Our Bible? A dark age in Continuity is a dark age in the human spirit. We need continuity to be more than animals.
With out Continuity we may exist in a state of perfect societal flow, and may even transcend into this singularity thing, but really what breaks down? Philosophy will fail? Reason spanning seasons will fail, and a prolonged desire to plan will fail. If you disagree with this, then why is it that the north is more productive than anywhere else in the world. Why is it that winter climates seem to evoke the need to be productive? Why do the Sweeds make the coolest lights? Necessity breads innovation.
Jared Diamond who has written Guns, Germs, and Steal, and Collapse seems to evoke the notion that having to plan for winter each year played a large role in human and human cultural development.
If technology moves so fast that planing and thinking about the future are fruitless, we are in chaos. We might even be without what we call humanity. We will be confined to the present, making decisions on instinct and gut checks rather than thoughtful discourse. We will be in a stress survival mode, which will remove us from higher thought. The poor are often challenged in this way, and study after study shows that the poor spend more money on the same activities. They spend more on food ( percentage wise ), more on transportation, etc... We should strive for the sort of wealth that allows us the pleasure to think and share.
If this erks you, it should. If you disagree, then you might need to check out the Long Now Foundation, which hopes to create a sense of continuity for us into the future.
I am beginning to think that John Nash's work ( a beautiful mind ) is more meaningful economically at a society level, than at a company a,b,c level. I think as a society we should no longer seek maximums. "Maximums are the root of too much evil, as they lack the room to account for exceptions." "A maximized system cannot leave room for the Black Swan, and are fragile by nature, asking to be toppled." John Nash's work suggests that in a competitive system one will find a more efficient use of resources going for 2nd best rather than the best. So, in the argument of sustainability, continuity, and stability; maximums undermine our ability's by overvaluing now when compared to the past and future.
As a society we need to to create an economics that has room for "grace".
Cancer Research, Blocking Cancer’s energy source, ATP, Lactic Acid, PEP, Pyruvate
Please take this at face value, and do your own investigation, but this looked like promising research.
If you put together these articles you might draw some
interesting conclusions.
- http://web.mit.edu/newsoffice/2010/cancer-metabolism-0917.html
- http://en.wikipedia.org/wiki/Pyruvic_acid
- http://en.wikipedia.org/wiki/Bromopyruvic_acid
- http://www.emphycorp.com/cancerprevention.html
- http://sportsmedicine.about.com/cs/exercisephysiology/a/aa053101a.htm
- http://www.cancer.gov/cancertopics/factsheet/prevention/physicalactivity
- http://www.usatoday.com/news/health/2009-08-12-breast-cancer_N.htm
This is what I gleamed from putting together the above sources.
It seems from these articles that cancer uses a cell division
technique that can be interrupted with a build up of lactic acid.
Lactic acid is more easily cleared up or removed which high sugar, or
carbohydrate diets. ( seems like it's good for runners, but not good
for cancer patients. )
It seems like cancer growth is at it's worst when Lactic acid has not
built up in a while to inhibit/ reset the rogue energy cycle that
forms.
Lactic acid builds up in muscles that are being used, but is
transported through blood flow to the rest of the body.
Blood flow during intense exercise is restricted to non-vital organs,
which would make it hard for lactic acid to reach those organs.
Prolonging Lactic acid in the body might help fight cancer.
From what I have read, this seems to indicated that intense exercise (
causing a warmness or burning in your muscles ) and not carb-loading
before a workout, or immediately afterwards is an ideal strategy. It
may be that the rise in cancer in our society might be related to the
sharp decline in intense exercise.
End of story, no pain no gain, when it comes to exercise.
It looks like they are looking for a pill to replace the simple need
for us to work physically harder in our daily lives.
Dinner Parties – Mad Scientist Edition
If you have ever thrown a dinner party, you might know how much work they are, but you also know how rewarding they are as a way to share time with friends, and to eat something good.
In San Francisco, I started a new weekly dinner party at my apartment in SoMa, and ever week we try to come up with a theme. Sometimes that theme is easy to pull off and other times it's a bit more of a hassle than it's worth.
This week I have been trying to pull stuff together for a mad scientist theme, and in a town like SF you would think there are ton's of lab supply stores. However, google has failed to easily find me any. Instead every biotech and chemtech startup seems to appear on the list rather than cool stores like 'American Science and Surplus' would back in milwaukee and chicago.
So for posterity sake I thought I would pass the ball forwards and document a few sites of interest to help prepare for the next 'Mad Scientist Themed-Dinner party'.
Links of aspiration and inspiration
- 'Dr. Horrible's Sing Along Blog'
- 'Dr. Steel's' Search for a utopian playland
- 'Beaker' from the Muppet
- 'The City of Lost Children'
- 'City of Ember' by Tim Robbins
Online Stores for beakers, flasks, funnels, petri dishes and test-tubes
- Science Gear
- Amazon - Pyrex Griffin Beakers
- Medixcorp - *requires IE
- Acme Vial and Glass Co.
- American Science and Surplus
Some Geek Chic-ery
A few XPath links to make YQL and HTML scraping easier.
quick and basic
http://www.w3schools.com/XPath/xpath_syntax.asp
great resource for detailed string functions
http://oreilly.com/catalog/xmlnut/chapter/ch09.html
quick list of all of the functions in xpath
http://www.w3schools.com/xpath/xpath_functions.asp
great example of more sophisticated function selectors
http://www.eggheadcafe.com/articles/20030627d.asp
multiple attribute
http://www.coderanch.com/t/128329/XML/XPATH-selection-based-multiple-attribute
fun with concat
http://blogs.sun.com/rajeshthekkadath/entry/xpath_searching_for_a_text
Who could buy Palm?
- Dell, Sony, HTC, etc.. -- Have mobile devices, but this could give them an edge.
- ASUS, Abit, etc.... -- All make hardware and could use a nice mobile platform.
- Amazon, Barns and Noble -- both make eReaders that could be decimated by the iPad, Palm's Web OS could give them the engineering talent along side a great OS to create a new Mobile media device.
- nVidia - if nVidia had a platform like WebOS, they could ensure a rockstar platform for their mobile chips.
- AMD - I don't think AMD has the cash to buy palm.
- IBM - Could be interesting, IBM has cash, and they don't have a mobile plan. This may be reason for IBM to go after a mobile leader.
- Oracle - Since Oracle has acquired Sun, MySQL, and .... OpenOffice by default. It might be interesting to see what this Tech Giant could do in the Mobile Enterprise Space.
- Google - Could buy Palm to strengthen it's position against Apple, and could port Palm's intellectual property to the Android OS.
- Nintendo - Hmmm, this could be interesting, as I think the companies would culturally work well together.
- Microsoft - Yeah, WM7 +++ MS could somehow make windows mobile better, or better arm it's lawyers against Apple. Can you imagine the MS Palm Courier?
Large Binary Numbers in Javascript
Have you ever wondered what the largest number is? At one point it was a googolplex, but unless you write a custom class to deal with really large numbers, your dev environment will never count to a googolplex.
For http://loc.is I am reworking a geohash encoding algorithm today, and I wanted to see what Javascript could handle. Strongly typed languages like Java have known upper limits, but languages like javascript are a bit more mysterious, so the only way to know is to find out. I wrote this function to test it, and I ran it in the firebug console on a random webpage ( firebug won't work unless the DOM is ready ).
for( i = 0 ; i < 500 ; i++){
var value = Math.pow(2,i) +1;
console.log ( i + "\n" +
value + "\n" +
value.toString(2));
}
The output finally looks like this, and reveals that 2^52 is the largest number you can add 1 to and still represent it as an integer. Any larger and javascript will just use it as a float, and you will not have a sufficient number of bits to represent it. I wonder if this is the same in all browser/ os combination pairs?
New jEdit Syntax Edit Modes for Perl+HTML & XML+CDATA + HTML
I love my syntax highlighting in jEdit, and why shouldn't I? Good Syntax Highlighting helps you catch code errors early. So about 2 years ago, I created a custom jEdit "Edit Mode" so that HTML buried in your XML CDATA tags would be Syntax Highlighted. Here is the original post on Google Groups ( http://groups.google.com/group/opensocial-api/browse_thread/thread/12e250246ab64054/343c858695e8eb12 ).
Here is the custom Edit Mode for XML->CDATA->HTML c-xml.xml
It uses a simple trick to identify the CDATA with HTML with a special CDATA tag like the one below
<![CDATA[<!--HTML-->
<html>
<!-- friendly neighborhood web page -->
</html>
]]>
The perl version is here perl-html.xml
So for the perl version i use the structure of Perl's HEREDOCS and use the following java capable regex
<<\p{Space}*(['"])([\p{Space}\p{Alnum}]*)\1;?\s*<\!--.*HTML.*-->
Which will HTML Syntax highlight any perl HEREDOC as HTML if it has <!-- HTML --> in it so the following should syntax highlight. There is however, one bug. If your HEREDOC is not well formed, meaning you have an unclosed tag css object, or javascript call, then the syntax highlighting gets stuck. Maybe I will figure out how to make this work for non well-formed HTML. any ideas?
print <<"EOT"; <!-- HTML -->
<html>
<!-- friendly neighborhood web page -->
</html>
EOT
Life changes you, things learned.
It's almost been 2 years since i moved to california and its been a rough adventurous ride, and I don't regret a moment of it. However, when I look back it's odd to see just how different I am today.
Since sundays are for reflection, here are a few bullet points.
- • living in milwaukee i felt very connected to the community, i knew so many people by name that everywhere i went was like an episode of cheers.
- • i used to drink and eat a lot more, today i eat 2/3rds to 1/2 as much food. it's so warm in california you really don't need much.
- • winter seems like someplace you visit, rather than live through.
- • i used to feel tech deprived, now it's easily accessible and to some degree less exotic
- • i understand better how some ideas are good ones and others are a waste of time
- • i now know i own too much stuff, too much ownership is a burden; you end up consuming more resources just keeping it and not appreciating it.
- • women like geeks and entrepreneurs here, i feel sexy here
- • hackathons used to be parties, now they seem like work
- • i miss bucketworks, but….. have found noisebridge, and techshop, but don't go enough.
- • owning cars seem silly, but i still love my 350z, but now i don't have a venue to enjoy it. i may just sell it and rent a sports car for a day, even at 500 dollars every few months, it's cheaper, than paying for insurance and parking.
- • in milwaukee, geeks were most easily found at goth industrial clubs, but out here, they are everywhere.
- • you live in a city, not in an apartment, neighborhoods are probably more important than your home.
- • being able to bike everywhere RULES.
- • public transit is better than no transit, but it still sucks, its a tragedy of the commons.
- • some graffiti is good, some is bad.
- • political parties are not relevant.
- • some people are so angry about somethings, that they will never listen about other things
- • what its like to have a lot of friends that are female, what it's like to have a female room mate; the secret is it's really not that different, except your guy friends probably are more likely to drink your beer and watch the same tav shows.
- • on average women think about sex way more than I would have thought 2 years ago.
- • women may think differently, but they think about the same things. ( ok, maybe not all of the same things, or maybe just not at the same level of detail )
- • the goal of any good conversation is to derive respect of and for someone regardless of weather you agree, find enjoyment in their fellowship, however not all people are respectable, and so don't spend your time with them
- • no matter how much you may want others to be independent and self capable, some people just want to follow, or lead. people may not be built for an egalitarian maximum.
- • any good piece of art, creates a deeper connection between people. it should make good conversation easy. ( see above )
Improving PHPBB2 and MySQL performance
For the last few days I have been trying to track down why Unity’s PHPBB 2 Forum (http://forum.unity3d.com) was so slow. Page rendering times were taking between 2 and 10 seconds and for me this was just unacceptable.
A proper website should be able to render a majority of the page in less than 250ms, and delivering most of the content in less than 50ms is ideal. Sites like Amazon, Yahoo, and Google have studied the effects of response time vs. features, and have found that response time is often times more important. Greg Linden pulls together a few sources on the topic in his blog.
So, 10 seconds was just unacceptable, and I was bound to find a solution. I started with the low hanging fruit, and installed PHP’s APC cache which is an OpCode Cache that stores compiled versions of the PHP code to reuse on the next request. There is evidence out there to suggest that not only do opcode caches reduce CPU requirements, but they also reduce your memory load. I saw a few sites that claim about a 3-4x performance increase on the CPU and about a 25% reduction in memory usage.
The next step, as the forums were still slow, was to start looking into our MySQL usage. Did the server have enough memory, were the MySQL caches and buffers large enough, or were searches and queries getting pruned to make room for more queries? After using MySQL’s GUI tools through an SSH Tunnel I was able to see that the server had 64MB of query cache, and only 90% of it was being used, so the caches were good.
Next was to look for slow queries, however a slow query or two would be hard to fix considering PHPBB was building the queries as part of the application and should have been tested in advance. If a slow query was the problem, it probably was not part of the application design, but rather an indication of something else that was going wrong. We did notice a few slow queries, but nothing ridiculous. However, what we did notice was that a number of queries were running slow because other ones where holding a lock for a long period. Hmmm?
Next we started looking at the problem from a system level using apps like ‘top’, ‘dstat’, ‘mytop’, etc… From here we found that disk IO was over working. Hmmm… but why?
I later found out that we were serving large files in our forum, however it did not seem like our forum was being used as a hot linking service, and I would suspect that PHPBB has some defense from that, but going forwards this is a concern of mine.
I finally found a post on the web talking about MySQL CPU spikes, and it was related to a corrupted table index. http://forums.fedoraforum.org/showthread.php?t=232008
After reading the forum for a bit, I thought, well there seems to be a few low risk commands I can run to look into our table integrity. Here is the full command list http://dev.mysql.com/doc/refman/5.0/en/mysqlcheck.html
First I ran
mysqlcheck -A -F -u [username] –p
This reported several table closed improperly errors, and took about 2-6 mins to run without taking down the forums.
Next I ran
mysqlcheck -A -q -r -u [username] –p
This is basically the fastest repair option, and would only fix minor corruption. It took about 5-10 min to run.
The result was that our MySQL usage dropped from 100-300% to 3-6% and page response times being about 200-500ms. I have a few other things I would like to try, from using our CDN with the forums to DB optimizations, to other server upgrades.
Over the weekend, I think I am going to run
mysqlcheck --all-databases --optimize -u [username] –p
Which should be more thorough, but might take an hour. This will defrag the table and resolve some indexing errors.