jdavid.net

If you have ever thrown a dinner party,  you might know how much work they are, but you also know how rewarding they are as a way to share time with friends, and to eat something good.

In San Francisco, I started a new weekly dinner party at my apartment in SoMa, and ever week we try to come up with a theme.  Sometimes that theme is easy to pull off and other times it’s a bit more of a hassle than it’s worth.

This week I have been trying to pull stuff together for a mad scientist theme, and in a town like SF you would think there are ton’s of lab supply stores.  However, google has failed to easily find me any.  Instead every biotech and chemtech startup seems to appear on the list rather than cool stores like ‘American Science and Surplus’ would back in milwaukee and chicago.

So for posterity sake I thought I would pass the ball forwards and document a few sites of interest to help prepare for the next ‘Mad Scientist Themed-Dinner party’.

Links of aspiration and inspiration

• ‘Dr. Horrible’s Sing Along Blog’
• ‘Dr. Steel’s’ Search for a utopian playland
• ‘Beaker’ from the Muppet
• ‘The City of Lost Children’
• ‘City of Ember’ by Tim Robbins

Online Stores for beakers, flasks, funnels, petri dishes and test-tubes

• Science Gear
• Amazon – Pyrex Griffin Beakers
• Medixcorp – *requires IE
• Acme Vial and Glass Co.
• American Science and Surplus

Some Geek Chic-ery

• Testtubes + Wedding = Way COOL!

quick and basic
http://www.w3schools.com/XPath/xpath_syntax.asp

great resource for detailed string functions
http://oreilly.com/catalog/xmlnut/chapter/ch09.html

quick list of all of the functions in xpath
http://www.w3schools.com/xpath/xpath_functions.asp

great example of more sophisticated function selectors
http://www.eggheadcafe.com/articles/20030627d.asp

multiple attribute
http://www.coderanch.com/t/128329/XML/XPATH-selection-based-multiple-attribute

fun with concat
http://blogs.sun.com/rajeshthekkadath/entry/xpath_searching_for_a_text

http://www.xml.com/pub/a/2002/08/14/xpath_tips.html?page=3

I thought, I would post this here, maybe someone will respond or google will link this.

Length of project: 1-2 weeks
Rate: DOE

Requirements:
• HTML, XHTML
• CSS2.1 and CSS3
• Experience with Photoshop or Illustrator ( or equivalent )
• Experience with Theme-ing wordpress, phpbb3, or mediawiki

Nice to Haves
• PHP5.2
• Javascript

This is a contract position, and the perfect candidate would have skinned at least a few to a few dozen websites, and is demonstrable. This is a short project, but we have a number of projects that come up.

About Unity Technologies
Unity Technologies is revolutionizing the game industry and was named one of the top five game companies of 2009 by Gamasutra after just four years on the market. Today, Unity Technologies has more than 120,000 registered users worldwide – including leading companies like Coca-Cola, Electronic Arts, Disney, LEGO, Mint.com, NASA and UPS, large and small studios, independent professionals, students and hobbyists – using the Unity platform to develop high-quality interactive 3D content for the web, mobile, and console. In addition to Web, PC, Mac, Wii™, Xbox 360, iPhone and iPad deployment, Unity Technologies has announced upcoming support for PS3 and Android. Unity Technologies is one of the fastest growing software companies and is aggressively innovating to expand usability, power and platform reach so that it can deliver on its vision of democratizing interactive 3D technology. Unity Technologies is headquartered in San Francisco and has development offices worldwide. For more information, visit unity3d.com.

Have you ever wondered what the largest number is?  At one point it was a googolplex, but unless you write a custom class to deal with really large numbers, your dev environment will never count to a googolplex.

For http://loc.is I am reworking a geohash encoding algorithm today, and I wanted to see what Javascript could handle.  Strongly typed languages like Java have known upper limits, but languages like javascript are a bit more mysterious, so the only way to know is to find out.  I wrote this function to test it, and I ran it in the firebug console on a random webpage ( firebug won’t work unless the DOM is ready ).

for( i = 0 ; i < 500 ; i++){

var value = Math.pow(2,i) +1;

console.log (   i + "\n" +

value + "\n" +

value.toString(2));

}

The output finally looks like this, and reveals that 2^52 is the largest number you can add 1 to and still represent it as an integer. Any larger and javascript will just use it as a float, and you will not have a sufficient number of bits to represent it.  I wonder if this is the same in all browser/ os combination pairs?

I love my syntax highlighting in jEdit, and why shouldn’t I?  Good Syntax Highlighting helps you catch code errors early.  So about 2 years ago, I created a custom jEdit “Edit Mode” so that HTML buried in your XML CDATA tags would be Syntax Highlighted.  Here is the original post on Google Groups ( http://groups.google.com/group/opensocial-api/browse_thread/thread/12e250246ab64054/343c858695e8eb12 ).

Here is the custom Edit Mode for XML->CDATA->HTML c-xml.xml

It uses a simple trick to identify the CDATA with HTML with a special CDATA tag like the one below

<![CDATA[<!--HTML-->  

<html>
<!-- friendly neighborhood web page -->
</html>

]]>

The perl version is here perl-html.xml

So for the perl version i use the structure of Perl’s HEREDOCS and use the following java capable regex

<<\p{Space}*(['"])([\p{Space}\p{Alnum}]*)\1;?\s*<\!--.*HTML.*-->

Which will HTML Syntax highlight any perl HEREDOC as HTML if it has <!– HTML –> in it so the following should syntax highlight.  There is however, one bug.  If your HEREDOC is not well formed, meaning you have an unclosed tag css object, or  javascript call, then the syntax highlighting gets stuck.  Maybe I will figure out how to make this work for non well-formed HTML.  any ideas?

print <<"EOT"; <!-- HTML -->

<html>
<!-- friendly neighborhood web page -->
</html>

EOT

It’s almost been 2 years since i moved to california and its been a rough adventurous ride, and I don’t regret a moment of it. However, when I look back it’s odd to see just how different I am today.

Since sundays are for reflection, here are a few bullet points.

• • living in milwaukee i felt very connected to the community, i knew so many people by name that everywhere i went was like an episode of cheers.
• • i used to drink and eat a lot more, today i eat 2/3rds to 1/2 as much food. it’s so warm in california you really don’t need much.
• • winter seems like someplace you visit, rather than live through.
• • i used to feel tech deprived, now it’s easily accessible and to some degree less exotic
• • i understand better how some ideas are good ones and others are a waste of time
• • i now know i own too much stuff, too much ownership is a burden; you end up consuming more resources just keeping it and not appreciating it.
• • women like geeks and entrepreneurs here, i feel sexy here
• • hackathons used to be parties, now they seem like work
• • i miss bucketworks, but….. have found noisebridge, and techshop, but don’t go enough.
• • owning cars seem silly, but i still love my 350z, but now i don’t have a venue to enjoy it. i may just sell it and rent a sports car for a day, even at 500 dollars every few months, it’s cheaper, than paying for insurance and parking.
• • in milwaukee, geeks were most easily found at goth industrial clubs, but out here, they are everywhere.
• • you live in a city, not in an apartment, neighborhoods are probably more important than your home.
• • being able to bike everywhere RULES.
• • public transit is better than no transit, but it still sucks, its a tragedy of the commons.
• • some graffiti is good, some is bad.
• • political parties are not relevant.
• • some people are so angry about somethings, that they will never listen about other things
• • what its like to have a lot of friends that are female, what it’s like to have a female room mate; the secret is it’s really not that different, except your guy friends probably are more likely to drink your beer and watch the same tav shows.
• • on average women think about sex way more than I would have thought 2 years ago.
• • women may think differently, but they think about the same things. ( ok, maybe not all of the same things, or maybe just not at the same level of detail )
• • the goal of any good conversation is to derive respect of and for someone regardless of weather you agree, find enjoyment in their fellowship, however not all people are respectable, and so don’t spend your time with them
• • no matter how much you may want others to be independent and self capable, some people just want to follow, or lead. people may not be built for an egalitarian maximum.
• • any good piece of art, creates a deeper connection between people. it should make good conversation easy. ( see above )

For the last few days I have been trying to track down why Unity’s PHPBB 2 Forum (http://forum.unity3d.com) was so slow. Page rendering times were taking between 2 and 10 seconds and for me this was just unacceptable.

A proper website should be able to render a majority of the page in less than 250ms, and delivering most of the content in less than 50ms is ideal. Sites like Amazon, Yahoo, and Google have studied the effects of response time vs. features, and have found that response time is often times more important. Greg Linden pulls together a few sources on the topic in his blog.

So, 10 seconds was just unacceptable, and I was bound to find a solution. I started with the low hanging fruit, and installed PHP’s APC cache which is an OpCode Cache that stores compiled versions of the PHP code to reuse on the next request. There is evidence out there to suggest that not only do opcode caches reduce CPU requirements, but they also reduce your memory load. I saw a few sites that claim about a 3-4x performance increase on the CPU and about a 25% reduction in memory usage.

The next step, as the forums were still slow, was to start looking into our MySQL usage. Did the server have enough memory, were the MySQL caches and buffers large enough, or were searches and queries getting pruned to make room for more queries? After using MySQL’s GUI tools through an SSH Tunnel I was able to see that the server had 64MB of query cache, and only 90% of it was being used, so the caches were good.

Next was to look for slow queries, however a slow query or two would be hard to fix considering PHPBB was building the queries as part of the application and should have been tested in advance. If a slow query was the problem, it probably was not part of the application design, but rather an indication of something else that was going wrong. We did notice a few slow queries, but nothing ridiculous. However, what we did notice was that a number of queries were running slow because other ones where holding a lock for a long period. Hmmm?

Next we started looking at the problem from a system level using apps like ‘top’, ‘dstat’, ‘mytop’, etc… From here we found that disk IO was over working. Hmmm… but why?

I later found out that we were serving large files in our forum, however it did not seem like our forum was being used as a hot linking service, and I would suspect that PHPBB has some defense from that, but going forwards this is a concern of mine.

I finally found a post on the web talking about MySQL CPU spikes, and it was related to a corrupted table index. http://forums.fedoraforum.org/showthread.php?t=232008

After reading the forum for a bit, I thought, well there seems to be a few low risk commands I can run to look into our table integrity. Here is the full command list http://dev.mysql.com/doc/refman/5.0/en/mysqlcheck.html

First I ran

mysqlcheck -A -F -u [username] –p

This reported several table closed improperly errors, and took about 2-6 mins to run without taking down the forums.

Next I ran

mysqlcheck -A -q -r -u [username] –p

This is basically the fastest repair option, and would only fix minor corruption. It took about 5-10 min to run.

The result was that our MySQL usage dropped from 100-300% to 3-6% and page response times being about 200-500ms. I have a few other things I would like to try, from using our CDN with the forums to DB optimizations, to other server upgrades.

Over the weekend, I think I am going to run

mysqlcheck --all-databases  --optimize -u [username] –p

Which should be more thorough, but might take an hour. This will defrag the table and resolve some indexing errors.

I finally have a new place, and it’s an open concept loft in San Francisco’s SOMA district. It’s got everything I have ever wanted in a place, but it’s got one problem, to afford it I had to take on a good friend as a roommate. I love having roommates in a traditional space, but having a roommate in an open concept loft could, well ,,,.. get bothersome in the future.

I am currently exploring and brainstorming concepts to create privacy. However, building a full height wall, although a feasible engineering solution, is not what I am looking for. We are looking for something that keeps the loft as open as possible, while allowing us to define private spaces. We have thought of the following options, but what ideas might you have?

• using plexiglass panels
• using triangular plexiglass panels and create an uneven surface
• using plexiglass with a negative pressure interior ( vacuum )
• using glass
• using foam
• hanging rugs
• hanging curtains
• using bloxes
• using a rubber skin around a substructure/ skeleton
• creating a plexiglass substructure
• creating a wooden substructure
• creating a cardboard substructure
• sucking out the air from with in the rubber encased structure
• doing something organic looking
• doing something modern
• using small bright light sources like LEDs to trick the eye of an on looker to contract and thus be unable to see passed the array of LEDs, one might call this a ‘light screen’
• use a light projector to overpower any refracting light in the room, thus creating optical privacy in one direction.
• using some sort of blinds

I will try to update this post with examples of interesting design at some point.  Maybe you can comment on a few.

Since this took me a little longer than I would have liked to find, and improve I thought I would post a little nugget about Linux Shell scripting and SVN that I learned today.

You can use the following line to add multiple files to your repository:

svn st | grep "^?" | awk '{ print $2}' | while read f; do svn add "$f"; done

However, the solution above from array studios to add multiple to a subversion repository fails when you have filenames that contain spaces. Normally files with spaces in them are not idea, however, I had a few files with spaces and I thought that there must be a solution. With a little digging, I found the AWK Manual very helpful. Below is an updated solution that should work with filenames that have spaces.

svn st | grep "^?" | awk '{ $1=""; print;}' | while read f; do svn add "$f"; done