So for Babelfin, the goal would be to list a set of phrases that one would want to learn, or that the user might enter, and then the plugin would pre-fetch the translation and store it in the language that the user wants to learn.  If the phrase list is short, using a key-value store like localStorage would make sense, as almost every browser supports it, including IE8, and even though there are small differences, the data model and pattern would still work. However for anything other than the minimal product, a larger data set will be required for more advanced users.  And as a result it would be nice to have a query-able data set.

Twice google has tried to solve this problem, once with Google Gears, and again with Web SQL Database, which works in Chrome, Opera, and Safari, but does not work in Firefox, or IE.  Microsoft and Mozilla do not want to add SQL as a language to browser side logic, and I happen to agree.

The result is IndexedDB, a javascript data storage model in the front end, however,  IndexedDB is not quite CouchDB or MongoDB, but rather a javascript level abstraction of storage data.  Several Mozilla folks seem to think that CouchDB might be built on top of it, and so there is BrowserCouch which will try to do just that.  But, where is the BrowserMongo ( Maybe we can call it Mango, as a Norwegian, told me that Mongo means retard in Norwegian, oops. )

So in any case there is still a gap for Browser Side Storage that is query-able and indexed.  I suppose something could be built on top of localStorage, but it would be a hack until the browsers fully support it.

I am trying to find out how much work it would be to put a stub wrapper on top of Webkit’s Web SQL Database layer and have it look like IndexedDB since that seems to be what Google will support eventually, once they figure out how to thread it, but it would be nice to have something now. ]]> http://jdavid.net/?feed=rss2&p=69 0 http://jdavid.net/?p=64 http://jdavid.net/?p=64#comments Mon, 26 Jul 2010 21:10:36 +0000 admin http://jdavid.net/?p=64 If you have ever thrown a dinner party,  you might know how much work they are, but you also know how rewarding they are as a way to share time with friends, and to eat something good.

In San Francisco, I started a new weekly dinner party at my apartment in SoMa, and ever week we try to come up with a theme.  Sometimes that theme is easy to pull off and other times it’s a bit more of a hassle than it’s worth.

This week I have been trying to pull stuff together for a mad scientist theme, and in a town like SF you would think there are ton’s of lab supply stores.  However, google has failed to easily find me any.  Instead every biotech and chemtech startup seems to appear on the list rather than cool stores like ‘American Science and Surplus’ would back in milwaukee and chicago.

So for posterity sake I thought I would pass the ball forwards and document a few sites of interest to help prepare for the next ‘Mad Scientist Themed-Dinner party’.

Links of aspiration and inspiration

• ‘Dr. Horrible’s Sing Along Blog’
• ‘Dr. Steel’s’ Search for a utopian playland
• ‘Beaker’ from the Muppet
• ‘The City of Lost Children’
• ‘City of Ember’ by Tim Robbins

Online Stores for beakers, flasks, funnels, petri dishes and test-tubes

• Science Gear
• Amazon – Pyrex Griffin Beakers
• Medixcorp – *requires IE
• Acme Vial and Glass Co.
• American Science and Surplus

Some Geek Chic-ery

• Testtubes + Wedding = Way COOL!

great resource for detailed string functions
http://oreilly.com/catalog/xmlnut/chapter/ch09.html

quick list of all of the functions in xpath
http://www.w3schools.com/xpath/xpath_functions.asp

great example of more sophisticated function selectors
http://www.eggheadcafe.com/articles/20030627d.asp

multiple attribute
http://www.coderanch.com/t/128329/XML/XPATH-selection-based-multiple-attribute

fun with concat
http://blogs.sun.com/rajeshthekkadath/entry/xpath_searching_for_a_text

http://www.xml.com/pub/a/2002/08/14/xpath_tips.html?page=3

Length of project: 1-2 weeks
Rate: DOE

Requirements:
• HTML, XHTML
• CSS2.1 and CSS3
• Experience with Photoshop or Illustrator ( or equivalent )
• Experience with Theme-ing wordpress, phpbb3, or mediawiki

Nice to Haves
• PHP5.2
• Javascript

This is a contract position, and the perfect candidate would have skinned at least a few to a few dozen websites, and is demonstrable. This is a short project, but we have a number of projects that come up.

About Unity Technologies
Unity Technologies is revolutionizing the game industry and was named one of the top five game companies of 2009 by Gamasutra after just four years on the market. Today, Unity Technologies has more than 120,000 registered users worldwide – including leading companies like Coca-Cola, Electronic Arts, Disney, LEGO, Mint.com, NASA and UPS, large and small studios, independent professionals, students and hobbyists – using the Unity platform to develop high-quality interactive 3D content for the web, mobile, and console. In addition to Web, PC, Mac, Wii™, Xbox 360, iPhone and iPad deployment, Unity Technologies has announced upcoming support for PS3 and Android. Unity Technologies is one of the fastest growing software companies and is aggressively innovating to expand usability, power and platform reach so that it can deliver on its vision of democratizing interactive 3D technology. Unity Technologies is headquartered in San Francisco and has development offices worldwide. For more information, visit unity3d.com.

For http://loc.is I am reworking a geohash encoding algorithm today, and I wanted to see what Javascript could handle.  Strongly typed languages like Java have known upper limits, but languages like javascript are a bit more mysterious, so the only way to know is to find out.  I wrote this function to test it, and I ran it in the firebug console on a random webpage ( firebug won’t work unless the DOM is ready ).

for( i = 0 ; i < 500 ; i++){

var value = Math.pow(2,i) +1;

console.log (   i + "\n" +

value + "\n" +

value.toString(2));

}

The output finally looks like this, and reveals that 2^52 is the largest number you can add 1 to and still represent it as an integer. Any larger and javascript will just use it as a float, and you will not have a sufficient number of bits to represent it.  I wonder if this is the same in all browser/ os combination pairs?

Here is the custom Edit Mode for XML->CDATA->HTML c-xml.xml

It uses a simple trick to identify the CDATA with HTML with a special CDATA tag like the one below

<![CDATA[<!--HTML-->  

<html>
<!-- friendly neighborhood web page -->
</html>

]]>

The perl version is here perl-html.xml

So for the perl version i use the structure of Perl’s HEREDOCS and use the following java capable regex

<<\p{Space}*(['"])([\p{Space}\p{Alnum}]*)\1;?\s*<\!--.*HTML.*-->

Which will HTML Syntax highlight any perl HEREDOC as HTML if it has <!– HTML –> in it so the following should syntax highlight.  There is however, one bug.  If your HEREDOC is not well formed, meaning you have an unclosed tag css object, or  javascript call, then the syntax highlighting gets stuck.  Maybe I will figure out how to make this work for non well-formed HTML.  any ideas?

print <<"EOT"; <!-- HTML -->

<html>
<!-- friendly neighborhood web page -->
</html>

EOT

Since sundays are for reflection, here are a few bullet points.

• • living in milwaukee i felt very connected to the community, i knew so many people by name that everywhere i went was like an episode of cheers.
• • i used to drink and eat a lot more, today i eat 2/3rds to 1/2 as much food. it’s so warm in california you really don’t need much.
• • winter seems like someplace you visit, rather than live through.
• • i used to feel tech deprived, now it’s easily accessible and to some degree less exotic
• • i understand better how some ideas are good ones and others are a waste of time
• • i now know i own too much stuff, too much ownership is a burden; you end up consuming more resources just keeping it and not appreciating it.
• • women like geeks and entrepreneurs here, i feel sexy here
• • hackathons used to be parties, now they seem like work
• • i miss bucketworks, but….. have found noisebridge, and techshop, but don’t go enough.
• • owning cars seem silly, but i still love my 350z, but now i don’t have a venue to enjoy it. i may just sell it and rent a sports car for a day, even at 500 dollars every few months, it’s cheaper, than paying for insurance and parking.
• • in milwaukee, geeks were most easily found at goth industrial clubs, but out here, they are everywhere.
• • you live in a city, not in an apartment, neighborhoods are probably more important than your home.
• • being able to bike everywhere RULES.
• • public transit is better than no transit, but it still sucks, its a tragedy of the commons.
• • some graffiti is good, some is bad.
• • political parties are not relevant.
• • some people are so angry about somethings, that they will never listen about other things
• • what its like to have a lot of friends that are female, what it’s like to have a female room mate; the secret is it’s really not that different, except your guy friends probably are more likely to drink your beer and watch the same tav shows.
• • on average women think about sex way more than I would have thought 2 years ago.
• • women may think differently, but they think about the same things. ( ok, maybe not all of the same things, or maybe just not at the same level of detail )
• • the goal of any good conversation is to derive respect of and for someone regardless of weather you agree, find enjoyment in their fellowship, however not all people are respectable, and so don’t spend your time with them
• • no matter how much you may want others to be independent and self capable, some people just want to follow, or lead. people may not be built for an egalitarian maximum.
• • any good piece of art, creates a deeper connection between people. it should make good conversation easy. ( see above )

A proper website should be able to render a majority of the page in less than 250ms, and delivering most of the content in less than 50ms is ideal. Sites like Amazon, Yahoo, and Google have studied the effects of response time vs. features, and have found that response time is often times more important. Greg Linden pulls together a few sources on the topic in his blog.

So, 10 seconds was just unacceptable, and I was bound to find a solution. I started with the low hanging fruit, and installed PHP’s APC cache which is an OpCode Cache that stores compiled versions of the PHP code to reuse on the next request. There is evidence out there to suggest that not only do opcode caches reduce CPU requirements, but they also reduce your memory load. I saw a few sites that claim about a 3-4x performance increase on the CPU and about a 25% reduction in memory usage.

The next step, as the forums were still slow, was to start looking into our MySQL usage. Did the server have enough memory, were the MySQL caches and buffers large enough, or were searches and queries getting pruned to make room for more queries? After using MySQL’s GUI tools through an SSH Tunnel I was able to see that the server had 64MB of query cache, and only 90% of it was being used, so the caches were good.

Next was to look for slow queries, however a slow query or two would be hard to fix considering PHPBB was building the queries as part of the application and should have been tested in advance. If a slow query was the problem, it probably was not part of the application design, but rather an indication of something else that was going wrong. We did notice a few slow queries, but nothing ridiculous. However, what we did notice was that a number of queries were running slow because other ones where holding a lock for a long period. Hmmm?

Next we started looking at the problem from a system level using apps like ‘top’, ‘dstat’, ‘mytop’, etc… From here we found that disk IO was over working. Hmmm… but why?

I later found out that we were serving large files in our forum, however it did not seem like our forum was being used as a hot linking service, and I would suspect that PHPBB has some defense from that, but going forwards this is a concern of mine.

I finally found a post on the web talking about MySQL CPU spikes, and it was related to a corrupted table index. http://forums.fedoraforum.org/showthread.php?t=232008

After reading the forum for a bit, I thought, well there seems to be a few low risk commands I can run to look into our table integrity. Here is the full command list http://dev.mysql.com/doc/refman/5.0/en/mysqlcheck.html

First I ran

mysqlcheck -A -F -u [username] –p

This reported several table closed improperly errors, and took about 2-6 mins to run without taking down the forums.

Next I ran

mysqlcheck -A -q -r -u [username] –p

This is basically the fastest repair option, and would only fix minor corruption. It took about 5-10 min to run.

The result was that our MySQL usage dropped from 100-300% to 3-6% and page response times being about 200-500ms. I have a few other things I would like to try, from using our CDN with the forums to DB optimizations, to other server upgrades.

Over the weekend, I think I am going to run

mysqlcheck --all-databases  --optimize -u [username] –p

Which should be more thorough, but might take an hour. This will defrag the table and resolve some indexing errors.