tag:blogger.com,1999:blog-58790505710987805012023-12-17T02:06:09.105+05:30The Code ArtistUnknownnoreply@blogger.comBlogger36125tag:blogger.com,1999:blog-5879050571098780501.post-63654451431206599362022-09-27T21:48:00.008+05:302023-02-02T16:47:24.244+05:30Cost-Effective Communication <p style="text-align: center;">Do you hate messaging someone for a quick clarification,<br />and waiting and praying that they see the message,<br />and then anxiously staring at "X is typing..." for several minutes ?<br /></p><p style="text-align: center;">Do you find it irritating to setup calls/meetings just to explain a tiny detail?<br /></p><p style="text-align: center;">In modern teams that believe in "Agile" and "Open Communication",<br />a common trope is...<br /></p><p></p><div style="text-align: center;"><blockquote>When someone gets stuck or needs help,<br /> then quick in-person discussions are so much nicer <br />and get the job done instantly.</blockquote></div><p></p><p style="text-align: justify;">While this sounds great at first sight, closer inspection reveals the massive overheads such an approach introduces. Imagine you are being invited for "quick discussions" by 3-4 people in your team whenever they get blocked every couple of times a day. Some of which often require multiple extended back-n-forth discussions. Your entire day is spent catering to their interruptions (though not "constant interruptions" for any of them individually, but a constant stream of interruptions for you.)<br /></p><p style="text-align: justify;">In such a scenario, a common technique to avoid being a bottleneck, is to have such discussions in <b>written down form, in wider/public chats, bug-trackers, task-trackers, wikis </b>(not direct-messages, not voice/video-calls, not face-to-face in-person chats).</p><p><br />...so that the folks who get stuck,</p><ul style="text-align: left;"><li>...can first search for previously "asked-and-answered" stuff<br />in the above wider/public channels of communication.<br /><br /></li><li>...and even if someone misses a previous discussion and approaches you,<br />you can now quickly point them to previous written down notes/comments/guides from your memory (or using search features of the communication tools).<br /></li></ul><p style="text-align: justify;">An added bonus of this approach is often you can find your own notes/guides in search results and thank your past self for saving 2 days of breaking your head trying to find a workaround for a weird corner-case which you have previously encountered and solved.<br /></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><img border="0" data-original-height="750" data-original-width="750" height="540" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyb9DO3VKLo023y29O-5lI4jM85fIQe4fzaNt2gkl_YCsWj54lbmWtotJ9VKgd1hhspEyuOh0fxRLCkJRPPFXIGKxQJgpKGYBIpAGOkhhsQD3T3K4UsuMxdZulfEEOQAjJBsBlyUrqU31VHD65ef_Lhjmi7EbHIujDPDKePNBiekEKmGs-yhXXFi9y/w640-h640/communication-modes.png" style="margin-left: auto; margin-right: auto;" width="540" /></td></tr><tr><td class="tr-caption" style="text-align: center;">(least-expensive) <b>A</b> < <b>B </b>< <b>C </b>< <b>D </b>(most-expensive)</td></tr></tbody></table><p></p><p>Meetings, Calls, Chats, Emails, Wikis, Documentation, ...<br />Various communication modes vary in their costs they impose on the participants.<br /></p><p>Costs include <u><b>time</b></u> and <u><b>effort</b></u> in the form of...<br /></p><ul style="text-align: left;"><li>duplicated/recurring communication that can be avoided.</li><li>disrupting the participants' schedule/flow.</li><li>fragmenting the participants' calendar</li><ul><li>reducing slots for deep/heads-down work.</li></ul></ul><p> </p><p>Common pitfalls that prevent efficient communication are:<br /></p><ul style="text-align: left;"><li>Using mode-<b>D </b>when mode-<b>B</b> or mode-<b>C</b> would suffice.</li><li>Not investing time upfront regularly to develop sufficient mode-<b>B</b>.</li><li>Not establishing clear expectations about "what is mode-<b>C</b> in the team".</li><li>Not teaching/learning how to be great at mode-<b>A</b>. </li></ul><p> </p><div style="text-align: center;">For an in-depth guide on <br />the principles and best-practices of asynchronous communication, <br />checkout <a href="https://about.gitlab.com/company/culture/all-remote/asynchronous" rel="nofollow" target="_blank">Gitlab's guide to Asynchronous Communication</a>.<br /></div><div><p></p></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5879050571098780501.post-47158841126970447272022-02-01T19:01:00.001+05:302022-02-01T19:01:22.557+05:30Tools to Fix a Perceived Lack of Leadership<p style="text-align: center;">Wondering how to be an influential servant-leader, and grow productive teams?</p><p style="text-align: center;">Did you notice certain teams repeatedly miss sprint-deadlines?</p><p style="text-align: center;">Do you sense that the team-members appear to lack motivation?</p><p style="text-align: center;">Here's a theory that can explain this (and a potential fix).
</p><div style="text-align: center;"><iframe allowfullscreen="" frameborder="0" height="360" marginheight="0" marginwidth="0" scrolling="no" src="//www.slideshare.net/slideshow/embed_code/key/emSU1Vi48XJxTb" style="max-width: 100%;" width="540"> </iframe></div>
Unknownnoreply@blogger.com0Bengaluru, Karnataka, India12.9715987 77.5945627-15.338635136178846 42.4383127 41.281832536178847 112.7508127tag:blogger.com,1999:blog-5879050571098780501.post-10527609737821842512022-01-30T20:01:00.004+05:302022-02-01T12:52:47.728+05:30The Impact of Alchian-Allen Effect on Software Development Teams<p style="text-align: justify;">While reading about <b><a href="https://byrnehobart.medium.com/the-30-year-mortgage-is-an-intrinsically-toxic-product-200c901746a" rel="nofollow" target="_blank">the toxicity of 30-year mortgages</a></b> this weekend,<br />i came across the <a href="https://en.wikipedia.org/wiki/Alchian%E2%80%93Allen_effect" target="_blank">Alchian-Allen Effect</a>.<br /></p><h2 style="text-align: left;">What is this Alchian-Allen effect?</h2><p>Imagine a fruit F grown in city-A.<br />Let's say the higher-quality of fruit F sells in city-A for $2,<br />whereas the lower-quality of fruit F sells in city-A for $1.</p><p></p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgfdUnjs1xL6jPZkKJCfBjrKaFck8BHwPAR7xB03XJ5CG6qSrIjDVNCeqWNXc3zXIH5fO8WeCdqUWoPJ_EKvOq0CDu2fcbQoOK2WTPdn1rvvu59U2cmWXynGJnnFOnjqQNJiSEUC5IZQoKXA_g16rIv9kgpIXxFbQGhMQhbcvPrr4BwUo61R5yw7MrI=s675" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="270" data-original-width="675" src="https://blogger.googleusercontent.com/img/a/AVvXsEgfdUnjs1xL6jPZkKJCfBjrKaFck8BHwPAR7xB03XJ5CG6qSrIjDVNCeqWNXc3zXIH5fO8WeCdqUWoPJ_EKvOq0CDu2fcbQoOK2WTPdn1rvvu59U2cmWXynGJnnFOnjqQNJiSEUC5IZQoKXA_g16rIv9kgpIXxFbQGhMQhbcvPrr4BwUo61R5yw7MrI" width="540" /></a></div>
<p></p><p style="text-align: justify;">Based on the fact that higher-quality of the fruit sells for twice the price of the lower-quality fruit, there would be a certain distribution of the consumption of both the varieties of the fruit. Based on the price-sensitivity of the people who buy the fruit in city-A, a certain percentage of people would buy a certain amount of the lower-quality fruit (as it is half-price of the more expensive variety).</p><p style="text-align: justify;">Now, think of a city-B that imports both the varieties of Fruit F. Let's say the transportation costs from city-A to city-B add $1 to the price of each fruit F. This is irrespective of the variety of the fruit being transported (after all the truck doesn't care how good the fruit is).</p><p>Assuming there's no price-gouging going on in city-B,<br />the higher-quality of fruit F sells in city-B for $3,<br />whereas the lower-quality of fruit F sells in city-B for $2.</p><p></p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhZ7VUsLYqg2tM6evEFevjgNJ4xdzXsgo4HD_ANZs6SksQFptd2Wae7Rvrk-8RzX2VzlnM-ej4Xi3bJWLXn263jmFoD_oOGx4oxbYF59aXNNZjCT7pK3Dh1j1GtpYSYUaOyPRWkpE8L19k4-c0640kPM1gPTfzQOHCaolzSkwTwu_NAtAMYisXgGoY3=s675" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="270" data-original-width="675" src="https://blogger.googleusercontent.com/img/a/AVvXsEhZ7VUsLYqg2tM6evEFevjgNJ4xdzXsgo4HD_ANZs6SksQFptd2Wae7Rvrk-8RzX2VzlnM-ej4Xi3bJWLXn263jmFoD_oOGx4oxbYF59aXNNZjCT7pK3Dh1j1GtpYSYUaOyPRWkpE8L19k4-c0640kPM1gPTfzQOHCaolzSkwTwu_NAtAMYisXgGoY3" width="540" /></a></div>
<p>Now, if the price-sensitivity of the people of city-B is similar to those in city-A,<br />then one would see a larger percentage of people in city-B buying the higher-quality fruit F in city-B.</p><p>The simple reason being,<br /></p><ul style="text-align: left;"><li>residents of city-A<br />need to pay a <span style="background-color: #fcff01;"><span style="color: red;">100% premium</span></span> for the better quality of fruit<br />($2 instead of $1),<br /></li></ul><ul style="text-align: left;"><li>residents of city-B<br />need to pay just a <span style="color: #38761d;"><span style="background-color: #fcff01;">50% premium</span></span> for the better quality of fruit<br />($3 instead of $2).</li></ul><p>An extreme result of the Alchian-Allen effect can be seen when the source of the additional fixed-cost has a limit. In the above example, if there is a limit to the amount of fruit that can be transported from city-A to city-B, then only the higher-quality fruit would be transported to city-B. This would result in city-A having all the lower-quality of the fruit (and if any higher-quality fruit was leftover due to the limit on transport to city-B).<br /></p><p></p><div style="text-align: justify;"><blockquote>NOTE: In both the individual scenarios above, the <a href="https://www.investopedia.com/terms/a/anchoring.asp" target="_blank">anchoring effect</a> appears to be at play as well. The Alchian-Allen effect can be seen as a composite of 2 instances of the anchoring effect with a common fixed-cost being the main differentiator between the 2 scenarios - ultimately leading to 2 different outcomes.<br /></blockquote></div><p></p><h2 style="text-align: left;">Alchian-Allen effect in Software-Development</h2><h4 style="text-align: left;">1. Buying Software vs. Building Software ?</h4><p style="text-align: justify;">Nowadays, most software teams that work of huge problems, are often posed the question - "Build or Buy". Whether to develop a necessary piece of software in-house or opt for such a software-solution from some other team/company.</p><p style="text-align: justify;">The Alchian-Allen effect may NOT be immediately obvious when choosing between Build vs. Buy, it tends to play a significant role over the long-run. Consider a team that opts for the "Build" option because the overall costs of developing the solution in-house are perceived to be lesser than the price demanded by existing solution providers. Such a team will in the future perceive the cost of each additional feature to be significantly higher (as anchored to the relatively lower cost of the initial deployment).</p><p style="text-align: justify;">Contrast this with a company that opts for the "Buy" option i.e. gets the software developed/maintained by another team/vendor, even though they perceive the initial deployment cost to be significantly higher. Such a team will in the future perceive the cost of each additional
feature to be significantly lower (as they are anchored to the relatively higher cost of the initial deployment).</p><p style="text-align: left;"></p><div style="text-align: justify;"><blockquote>Can this explain, why sometimes Non-Commercial Open-Source Software is criticized due to the costs associated with maintaining it over the long-run? Whereas such costs associated with similar proprietary software are written-off as "just the cost of doing business".</blockquote></div><p></p><h4 style="text-align: left;">2. Grooming/Training within vs. Hiring from the outside ?<br /></h4><p style="text-align: left;">Let us compare 2 situations...</p><p style="text-align: justify;"><b>Scenario A</b> - An employer is asked to invest in an existing employee. This could be in the form of a paid training/certification, or even some dedicated time-off for the employee to focus on learning/improvement, and other such secondary activities. Let's say the cost of such grooming/training an existing employee is $X.<br /></p><p style="text-align: justify;"><b>Scenario B</b> - An employer in considering hiring a new employee. There are 2 potential candidates. The difference between the 2 being that one of them has had the additional relevant training as part their previous work and is also demanding a slightly higher pay (let's say the same additional amount of $X) compared to the other candidate.<br /></p><p style="text-align: left;">Note that both the above scenarios involve the employer choosing between...<br /></p><ol style="text-align: left;"><li>An employee with a certain set of skills.</li><li>An employee with a certain larger set of skills that, and costs $X more.<br /></li></ol><p style="text-align: justify;">It sounds plausible that, when anchored with the relatively larger costs associated with hiring a new employee, a typical employer would view the amount $X as "a good deal", and is thus more likely to choose better employees when hiring. Whereas, when viewed in the context of a relatively smaller cost to the company of an existing employee, given the choice of a similar cost of training/grooming $X, they may view the same $X as too high a price to pay. </p><p style="text-align: left;">If this sounds like your team,<br />then the Alchain-Allen effect appears to be biasing you <br />towards better external hires than better grooming within the team.<br /></p>Unknownnoreply@blogger.com0Bengaluru, Karnataka, India12.9715987 77.5945627-15.338635136178846 42.4383127 41.281832536178847 112.7508127tag:blogger.com,1999:blog-5879050571098780501.post-61260859456708372022-01-29T16:36:00.001+05:302022-01-29T16:36:56.844+05:30Watch Bonds traded on NSE India - Part1On the website of the National Stock Exchange (NSE) India, the details of various Bonds traded on the exchange are published. Anyone can view this publicly available data...
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgR0KHmZgc7gGy-AO6DnfjSz1Ovc93CxYAueSfoFGkUVgYlmzg0QN7k9HxW-fJURd7MBqXfUELXdTpmzpL2qaNNOcXpmf_QafTWuIGWq9NLgxLA_NIrbtHO_ezNNsw0CiT3U2GBA_RRYp0gM9NpiNXgxsL0AVzhGFWIg6V6lp9xJHAMErTfviFYx-PV=s1280" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="800" data-original-width="1280" src="https://blogger.googleusercontent.com/img/a/AVvXsEgR0KHmZgc7gGy-AO6DnfjSz1Ovc93CxYAueSfoFGkUVgYlmzg0QN7k9HxW-fJURd7MBqXfUELXdTpmzpL2qaNNOcXpmf_QafTWuIGWq9NLgxLA_NIrbtHO_ezNNsw0CiT3U2GBA_RRYp0gM9NpiNXgxsL0AVzhGFWIg6V6lp9xJHAMErTfviFYx-PV=s540" width="540" /></a></div>
...on <b><a href="https://www.nseindia.com/market-data/bonds-traded-in-capital-market" target="_blank">this page</a></b> which is updated in real-time (or near real-time).<br />Sufficient to keep an eye on numerous bonds traded on the exchange.
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEg9ZlokEZ410kcw_rlyF4xtFkVIlvoQxbjJaRWHdUOJBHFfgXhuC6cuL8RjeuBu9Byck39HE8rz7ZmWLhCqE0Swct6His7hSVejRnbvsiH6DsoR4VVLbNuqt6sAIPcAFpz3Rr7LU2cxD0b3QVkxiLDa28EirfCwCrMxgKCriVyIb0GbG08_hNlLkwgm=s1280" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="800" data-original-width="1280" src="https://blogger.googleusercontent.com/img/a/AVvXsEg9ZlokEZ410kcw_rlyF4xtFkVIlvoQxbjJaRWHdUOJBHFfgXhuC6cuL8RjeuBu9Byck39HE8rz7ZmWLhCqE0Swct6His7hSVejRnbvsiH6DsoR4VVLbNuqt6sAIPcAFpz3Rr7LU2cxD0b3QVkxiLDa28EirfCwCrMxgKCriVyIb0GbG08_hNlLkwgm=s540" width="540" /></a></div>
<p style="text-align: justify;">However, unlike the stock of a company that trades under single symbol on the exchange, bonds of a single company often have multiple series, each with their different yields and maturity-dates. Thus, even if one is interested in a single company, often one would need to monitor dozens of bonds of the same company to find one that has the desired yield and is also currently being traded.</p><p>To simplify this workflow, we can scrape the above NSE webpage and filter the bonds of interest to us. Here's one way we can do that using simple Python code...
<script src="https://gist.github.com/TheCodeArtist/0e81603488b927c1cf6341209e12f78e.js"></script>
</p><p>Running the above script <b>nse-bond-watch.py<br /></b> on a system that has access to the internet,<br />we receive the <a href="https://gist.github.com/TheCodeArtist/18202faa047c0c10bf2c850e952464a6" target="_blank"> <b>data in CSV and JSON formats</b></a>.</p><div class="separator" style="clear: both; text-align: center;"></div>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiSa8wCFZdTOD1PyERGWEuENM-xIgerlxR1BdsyiuZXkzTXJ_z9ILsN32qE4Vcv6WAdjX75jalgR5NHrq6UIEtT2Kow5hDDeN4pdTgyae0OgEGAjIWzR9IagmXzdcZZdVE17v2A7FZcFPXp4XeJEeHI7GJDjHjy3GL5LxuGjKvOOP5o8ouUMmH6B4c8=s1000" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="500" data-original-width="1000" height="160" src="https://blogger.googleusercontent.com/img/a/AVvXsEiSa8wCFZdTOD1PyERGWEuENM-xIgerlxR1BdsyiuZXkzTXJ_z9ILsN32qE4Vcv6WAdjX75jalgR5NHrq6UIEtT2Kow5hDDeN4pdTgyae0OgEGAjIWzR9IagmXzdcZZdVE17v2A7FZcFPXp4XeJEeHI7GJDjHjy3GL5LxuGjKvOOP5o8ouUMmH6B4c8=s540" width="540" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgNKw0FSa_zCEqLxPw4YnlCw_9BKw9AIqwuNEm29NSMW3ptRk_FxV9qkDeXR8dO2ZsZ52rW8KeKE0RjuhTVUCAQpSyO8s5Gy8eJ9yzJSkHTh7H9SJeCc5CHbKSdMmZW9JHw82DUC6cVeLFa1pFiSpdIZxn_80aPC61GeaBnSH9ejdwxeDfUNAyJBKP-=s1000" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="500" data-original-width="1000" height="160" src="https://blogger.googleusercontent.com/img/a/AVvXsEgNKw0FSa_zCEqLxPw4YnlCw_9BKw9AIqwuNEm29NSMW3ptRk_FxV9qkDeXR8dO2ZsZ52rW8KeKE0RjuhTVUCAQpSyO8s5Gy8eJ9yzJSkHTh7H9SJeCc5CHbKSdMmZW9JHw82DUC6cVeLFa1pFiSpdIZxn_80aPC61GeaBnSH9ejdwxeDfUNAyJBKP-=s540" width="540" /></a></div>
<br />
<p>In subsequent posts in this series,<br />
we will explore how we can maintain this data locally, as well as filter it.
</p>Unknownnoreply@blogger.com0Bengaluru, Karnataka, India12.9715987 77.5945627-15.338635136178846 42.4383127 41.281832536178847 112.7508127tag:blogger.com,1999:blog-5879050571098780501.post-90387256130968975432019-02-03T13:06:00.000+05:302019-02-03T14:19:43.640+05:30Commoditise the Supply-Chain, Improve your Life<div style="text-align: center;">
<div style="text-align: left;">
<span style="font-family: "trebuchet ms" , sans-serif;"></span></div>
<div style="text-align: center;">
<div style="text-align: left;">
<span style="font-family: "trebuchet ms" , sans-serif;">What do the following have in common?…</span></div>
<span style="font-family: "trebuchet ms" , sans-serif;">
</span>
<ul style="text-align: left;">
<li><span style="font-family: "trebuchet ms" , sans-serif;">Microsoft in the ’90s</span></li>
<li><span style="font-family: "trebuchet ms" , sans-serif;">Google in the ’00s</span></li>
<li><span style="font-family: "trebuchet ms" , sans-serif;">Amazon in the ’10s</span></li>
<li><span style="font-family: "trebuchet ms" , sans-serif;">Atlassian (makers of JIRA, Confluence)</span></li>
<li><span style="font-family: "trebuchet ms" , sans-serif;">Uber</span></li>
<li><span style="font-family: "trebuchet ms" , sans-serif;">Elon Musk</span></li>
</ul>
<blockquote class="tr_bq">
<div style="text-align: left;">
<b><span style="font-family: "trebuchet ms" , sans-serif;">They all successfully commoditised the supply chain in their domain.</span></b></div>
</blockquote>
<br />
<div style="text-align: justify;">
<span style="font-family: "trebuchet ms" , sans-serif;"><i><b>Commoditising the Supply Chain</b></i> is a simple, yet powerful approach to
disrupt an existing status-quo. Numerous companies have successfully
employed this strategy in various industries/domains.</span></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<span style="font-family: "trebuchet ms" , sans-serif;">
</span></div>
<div style="text-align: justify;">
<span style="font-family: "trebuchet ms" , sans-serif;">This talk will briefly go over real-life examples that demonstrate this technique.</span></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<span style="font-family: "trebuchet ms" , sans-serif;">
</span></div>
<div style="text-align: justify;">
<span style="font-family: "trebuchet ms" , sans-serif;">We will then discuss how this technique can be applied by individuals
when confronted with seemingly insurmountable challenges in daily life.
How this can be applied to counter bullies at school, learn that new
language, get that promotion at work, and everything in between.</span></div>
<div style="text-align: justify;">
</div>
<span style="font-family: "trebuchet ms" , sans-serif;">
</span>
<br />
<blockquote>
<div style="text-align: left;">
<span style="font-family: "trebuchet ms" , sans-serif;"><b>Commoditise the Supply-Chain<br />
</b>and never worry about having to force yourself to do what you don’t want to.</span></div>
</blockquote>
<br />
<span style="font-family: "trebuchet ms" , sans-serif;">Slides from the talk i gave at <b>BarCamp Bangalore Spring 2019</b></span></div>
<span style="font-family: "trebuchet ms" , sans-serif;"><br /><iframe allowfullscreen="" frameborder="0" height="414" marginheight="0" marginwidth="0" scrolling="no" src="//www.slideshare.net/slideshow/embed_code/key/DJIWVECf2bEwdF" style="border-width: 1px; border: 1px solid #ccc; max-width: 100%;" width="500"> </iframe></span> </div>
<div style="margin-bottom: 5px; text-align: center;">
<span style="font-family: "trebuchet ms" , sans-serif;"><b> </b></span><br />
<span style="font-family: "trebuchet ms" , sans-serif;"><b><a href="https://www.slideshare.net/cvs26/improve-your-life-by-commoditising-the-supplychain" target="_blank" title="Improve your Life by Commoditising the Supply Chain">Improve your Life by Commoditising the Supply Chain</a></b></span> </div>
Unknownnoreply@blogger.com3Bengaluru, Karnataka, India12.9715987 77.59456269999998312.4764182 76.949115699999979 13.4667792 78.240009699999987tag:blogger.com,1999:blog-5879050571098780501.post-64927687564172567332018-06-10T23:20:00.000+05:302018-06-10T23:51:22.902+05:30Bottom Halves on Linux<div style="text-align: center;">
<span style="font-family: "trebuchet ms" , sans-serif;"><br /><iframe allowfullscreen="" frameborder="0" height="320" marginheight="0" marginwidth="0" scrolling="no" src="//www.slideshare.net/slideshow/embed_code/key/dbKtPMBakESl3" style="border-width: 1px; border: 1px solid #ccc; max-width: 100%;" width="500"> </iframe></span> </div>
<div style="margin-bottom: 5px; text-align: center;">
<span style="font-family: "trebuchet ms" , sans-serif;"><b> <a href="https://www.slideshare.net/cvs26/bottom-halves-on-linux" target="_blank" title="Bottom halves on Linux">Bottom halves on Linux</a></b></span> </div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5879050571098780501.post-36464018189668944762017-07-29T16:34:00.001+05:302020-01-19T16:21:44.320+05:30Optimise project scheduling using CD3<div style="text-align: justify;">
</div>
<div style="text-align: center;">
How to measure value? CD3!<br />
<br />
Optimise project scheduling using <b>CD3 - <u>C</u>ost of <u>D</u>elay, <u>D</u>ivided by <u>D</u>uration</b><br />
(a.k.a. WSJF - Weighted Shortest Job First)</div>
<div style="text-align: justify;">
<br />
Release-planning, sprint-planning, task-scheduling - all require a metric to prioritise between mutiple activities. A simple strategy like higher-business-values-tasks-first does NOT guarantee the most optimal schedule for a team/organisation.</div>
<br />
<blockquote class="tr_bq">
Cost of Delay, Divided by Duration i.e. CD3 is a metric that:<br />
- is easy to understand,<br />
- simple to calculate,<br />
- enables a more effective strategy.</blockquote>
<br />
<div style="text-align: center;">
<iframe allowfullscreen="" frameborder="0" height="330" marginheight="0" marginwidth="0" scrolling="no" src="https://www.slideshare.net/slideshow/embed_code/key/sDyfvy81fJyK6u" style="border-width: 1px; border: 1px solid #CCC; max-width: 100%;" width="510"> </iframe> </div>
<br />
<div style="text-align: center;">
Download the complete slide-deck - <a href="https://www.slideshare.net/cvs26/effort-estimation-73647698" target="_blank"><b>CD3 – Cost of Delay, Divided by Duration</b></a>.<br />
<br />
How to get the "<b>duration</b>" right? - Checkout this slide-deck on <b><a href="https://thecodeartist.blogspot.com/2017/04/effort-estimation.html" target="_blank">Effort Estimation</a></b>.</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5879050571098780501.post-47123768007780671852017-04-11T15:09:00.000+05:302018-02-19T21:21:51.692+05:30Effort Estimation<div style="text-align: center;">
<a href="https://youtu.be/0-VGipyKJ94?t=13s" target="_blank">[VIDEO] Importance of Effort Estimation and how to get it right. </a><br />
<br /> A talk on Effort Estimation <br />at ScrumBLR 19th Meetup<br />
and BarCamp Bangalore Spring 2018 </div>
<b><br /></b>
<br />
<div style="text-align: center;">
<b>6 REASONS why Estimates tend to be wrong!
</b></div>
<br />
<div style="text-align: center;">
<iframe allowfullscreen="" frameborder="0" height="330" marginheight="0" marginwidth="0" scrolling="no" src="//www.slideshare.net/slideshow/embed_code/73647698" style="border-width: 1px; border: 1px solid #CCC; max-width: 100%;" width="510"> </iframe> </div>
<br />
<div style="text-align: center;">
<blockquote class="tr_bq">
<div style="text-align: center;">
A newer updated version of the complete slide-deck<br />
is now available for free (CC-BY-SA).<br />
<br />
Download <a href="https://goo.gl/YnyoG4" target="_blank"><b>Effort Estimation</b></a> v2.
</div>
</blockquote>
</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5879050571098780501.post-84729317219477413202015-11-30T01:09:00.003+05:302015-12-10T10:19:06.850+05:30git gc, loose and packed git objects<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script>
<span style="font-size: large;">Intro:</span><br />
Recently during lunch, a bunch of us started discussing git internals. One popular point of contention that came-up was how git stores the incremental changes internally?<br />
<br />
<span style="font-size: large;">The Question:</span><br />
<blockquote class="tr_bq">
Does the <span style="font-family: "courier new" , "courier" , monospace;"><b>.git</b></span> subdirectory contain:<br />
- compressed diffs that apply on the original version of each file?<br />
<u><i>OR</i></u><br />
- compressed copy of each version of each modified file? </blockquote>
<span style="font-size: large;">The Answer:</span><br />
Different blogs appeared to claim differently [<b><a href="https://codewords.recurse.com/issues/two/git-from-the-inside-out" rel="nofollow" target="_blank">1</a></b>] [<a href="https://schacon.github.io/gitbook/1_the_git_object_model.html" rel="nofollow" target="_blank"><b>2</b></a>], and we were too lazy to go look-up the definitive source - <a href="http://git.kernel.org/cgit/git/git.git/" target="_blank">the source-code of git</a>.<br />
<br />
<div style="text-align: center;">
Warning: The following video is <span class="st"><i>processed in a facility</i> that also processes <i>nuts.</i></span></div>
<div style="text-align: center;">
<span class="st"><i> </i></span>Contains traces of the <i><a href="http://www.imdb.com/title/tt0059968/" rel="nofollow" target="_blank">60's Batman TV show</a></i>.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/rVP6w6WMxGk/0.jpg" frameborder="0" height="270" src="https://www.youtube.com/embed/rVP6w6WMxGk?feature=player_embedded" width="480"></iframe></div>
<embed here="" screencast="" video="" youtube=""></embed><br />
<span style="font-size: large;">The Conclusion:</span><br />
<br />
Another mystery solved. git gc to the rescue...<br />
<br />
<span style="font-size: large;">References:</span><br />
[1] <a href="https://codewords.recurse.com/issues/two/git-from-the-inside-out" rel="nofollow" target="_blank">https://codewords.recurse.com/issues/two/git-from-the-inside-out</a><br />
[2] <a href="https://schacon.github.io/gitbook/1_the_git_object_model.html" rel="nofollow" target="_blank">https://schacon.github.io/gitbook/1_the_git_object_model.html</a><br />
[3] <a href="https://schacon.github.io/gitbook/7_how_git_stores_objects.html" rel="nofollow" target="_blank">https://schacon.github.io/gitbook/7_how_git_stores_objects.html</a><br />
[4] <a href="https://schacon.github.io/gitbook/7_the_packfile.html" rel="nofollow" target="_blank">https://schacon.github.io/gitbook/7_the_packfile.html</a><br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-C30v8RmDxak/Vlm2pjcvq_I/AAAAAAAACM4/ZADdOQ_g_yI/s1600/to-the-garbage-dump.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="380" src="http://1.bp.blogspot.com/-C30v8RmDxak/Vlm2pjcvq_I/AAAAAAAACM4/ZADdOQ_g_yI/s500/to-the-garbage-dump.png" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Meanwhile...</b> having solved the riddle,<br />
the caped-crusader and the wonder-boy went
to Gotham city's garbage collection facility<br />
in the Bat-mobile to foil
whatever plan the Riddler's hatching...</td></tr>
</tbody></table>
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5879050571098780501.post-19315197050589932062015-04-30T23:41:00.000+05:302016-11-23T14:58:14.347+05:30Git previsionNeed to quickly check a previous version of a specific file?...<br />
Do not want to switch the entire repository to an older commit and back?...<br />
<br />
<blockquote class="tr_bq">
A combination of git aliases/awk/shell-functions to the rescue</blockquote>
Here is a quick and intuitive way to <a href="https://gist.github.com/TheCodeArtist/a90978ebca0ff6743036" target="_blank">checkout a older version of a single file in git</a>.<br />
<br />
<script src="https://gist.github.com/TheCodeArtist/a90978ebca0ff6743036.js"></script>
Basically the command does a <span style="font-family: "courier new" , "courier" , monospace;"><b>git log</b></span> on the specified file and picks the appropriate <span style="font-family: "courier new" , "courier" , monospace;"><b>commit-id</b></span> in the history of the file and then runs <span style="font-family: "courier new" , "courier" , monospace;"><b>git checkout</b></span> to the <b><span style="font-family: "courier new" , "courier" , monospace;">commit-id</span></b> for the specified <span style="font-family: "courier new" , "courier" , monospace;"><b>file</b></span>.<br />
<br />
Essentially, all that one would manually do in this situation,
wrapped-up in one beautiful, efficient git-alias - <a href="https://gist.github.com/TheCodeArtist/a90978ebca0ff6743036#file-git-prevision" rel="nofollow" target="_blank"><span style="font-family: "courier new" , "courier" , monospace;"><b>git-prevision</b></span></a> <br />
<div style="text-align: center;">
<blockquote class="tr_bq">
Liked git-prevision? Help others discover git-prevision.<br />
<a href="http://stackoverflow.com/a/29980518/319204" target="_blank"><b>Upvote <span style="font-family: "courier new" , "courier" , monospace;">git-prevision</span> on StackOverflow</b>.</a>
</blockquote>
</div>
Unknownnoreply@blogger.com0DadaMastan Layout, Manayata Tech Park, Nagavara, Bengaluru, Karnataka, India13.047205027169705 77.619073390960713.045271527169705 77.6165518909607 13.049138527169704 77.6215948909607tag:blogger.com,1999:blog-5879050571098780501.post-51368341231109374532015-04-14T22:15:00.000+05:302017-05-10T15:29:53.516+05:30Use-Case is Everything<div style="text-align: justify;">
The importance of a proper requirements gathering process, including detailed use-cases at the beginning of the software development process is often underestimated.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Common problems like feature-bloat, schedule-overruns, customer
dissatisfaction are easily avoided by mandating during the requirements
gathering stage, the preparation of an artifact containing a
comprehensive list of detailed use-cases of the final system.</div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<iframe allowfullscreen="" frameborder="0" height="420" marginheight="0" marginwidth="0" scrolling="no" src="//www.slideshare.net/slideshow/embed_code/46889279" style="border-width: 1px; border: 1px solid #CCC; margin-bottom: 5px; max-width: 100%;" width="510"> </iframe> </div>
<div style="text-align: justify;">
<br />
<div style="text-align: center;">
Download the complete slide-deck - <a href="http://www.slideshare.net/cvs26/usecase-is-everything" target="_blank"><b>Use-Case is Everything</b></a>.</div>
</div>
<div style="margin-bottom: 5px;">
<b> </b><b><a href="https://www.slideshare.net/cvs26" target="_blank"></a></b> <br />
<b> </b><b><a href="https://www.slideshare.net/cvs26" target="_blank"></a></b> </div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5879050571098780501.post-45879679136178276012015-01-18T22:20:00.003+05:302015-01-18T22:31:20.973+05:30MAX_JIFFY_OFFSET: msleep() beyond infinityEver wondered what would happen if a driver in the Linux kernel were to invoke Buzz Lightyear's oft-quoted catchphrase?<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-L0OlV9kgqfE/VLvVZ8oICBI/AAAAAAAAB-w/DMxfTpblthE/s1600/beyond-infinity.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-L0OlV9kgqfE/VLvVZ8oICBI/AAAAAAAAB-w/DMxfTpblthE/s1600/beyond-infinity.jpg" width="500" /></a></div>
<br />
If not then today is your lucky day, you can ponder over this million-dollar(price of Lightyear's left wing-nut :P) question safe in the comfort of the fact that the definitive answer is only as far as a couple of clicks of your mouse-wheel.<br />
<br />
<blockquote>
<b>NOTE</b>: This article discusses the intricacies of the implementation of <span style="font-family: "Courier New",Courier,monospace;"><b>msleep()</b></span>, specifically the conversion between milliseconds and jiffies within the Linux kernel. For details on implementing blocking calls and other such constructs that wait indefinitely, refer to <i>Section 6.2 Blocking I/O of LDD3</i> <a href="http://www.makelinux.net/ldd3/chp-6-sect-2" rel="nofollow" target="_blank">here</a> or <a href="http://lwn.net/images/pdf/LDD3/ch06.pdf" target="_blank">here</a>.</blockquote>
<br />
First a few numerical constants relevant to this discussion<br />
<span style="font-family: "Courier New",Courier,monospace;">Largest unsigned 32bit number = <b>ULONG_MAX</b> = 0xFFFFFFFF<br />Largest signed 32bit number = <b>LONG_MAX</b> = 0x7FFFFFFF<br />Largest jiffies defined = <b>MAX_JIFFY_OFFSET</b> = 0x3FFFFFFE</span><br />
<br />
A "jiffy" is defined as 1/HZ seconds. On a typical system with HZ=100, 1Jiffy works out to be 10ms in terms of real world units of time.<br />
<br />
<div style="text-align: center;">
MAX_JIFFY_OFFSET = 0x3FFFFFFE jiffies</div>
<div style="text-align: center;">
= 1073741822jiffies</div>
<div style="text-align: center;">
= ~10737418seconds</div>
<div style="text-align: center;">
= ~2982 hrs</div>
<div style="text-align: center;">
= ~124days</div>
<div style="text-align: center;">
= ~17weeks</div>
<br />
<span style="font-family: "Courier New",Courier,monospace;">msleep()</span> uses<span style="font-family: "Courier New",Courier,monospace;"> msecs_to_jiffies()</span>, which relies on <span style="font-family: "Courier New",Courier,monospace;">MAX_JIFFY_OFFSET</span> as the definition of <a href="http://lxr.free-electrons.com/source/kernel/time/time.c#L505" target="_blank">infinite sleep timeout</a>. <br />
<br />
Now consider the following input/output map of the <span style="font-family: "Courier New",Courier,monospace;">msec_to_jiffies()</span> for the entire range of <b>32bit unsigned int</b>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-z3fMBTGuda8/VLvZR3WyV0I/AAAAAAAAB-8/lFWfKz0Gmco/s1600/msecs_to_jiffies.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-z3fMBTGuda8/VLvZR3WyV0I/AAAAAAAAB-8/lFWfKz0Gmco/s1600/msecs_to_jiffies.png" width="500" /></a></div>
<br />
<br />
As is apparent, msecs-jiffies mapping is mostly linearly. Starting from 0, for increasing values of input, msec_to_jiffies() returns correspondingly larger values. However once the input msecs exceeds LONG_MAX, the output jiffies is clamped to MAX_JIFFY_OFFSET.<br />
<br />
Its obvious that, for certain values of input to msecs_to_jiffies()
the output result exceeds its definition of "infinity" i.e. exceeds
MAX_JIFFY_OFFSET. In fact, this happens for a quarter of the range of
32bit unsigned int! <a href="https://www.youtube.com/watch?v=-vohNUTTx3A" target="_blank">Pretty brain-dead, you say, eh?</a><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://www.youtube.com/watch?v=-vohNUTTx3A" target="_blank"><img alt="https://www.youtube.com/watch?v=-vohNUTTx3A" border="0" src="http://3.bp.blogspot.com/-4A6FsdQMKto/VLvaXsysU-I/AAAAAAAAB_I/2HHcqr_BisU/s1600/whyTrillions.jpg" /></a></div>
<br />
To be fair, the patch that introduced this definition of MAX_JIFFY_OFFSET is perfectly fine as it solves a genuine practical problem.<br />
<span style="font-family: "Courier New",Courier,monospace;"><a href="http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?id=9f907c0144496e464bd5ed5a99a51227d63a9c0b" target="_blank">commit 9f907c0144496e464bd5ed5a99a51227d63a9c0b</a></span><br />
<span style="font-family: "Courier New",Courier,monospace;">Author: Ingo Molnar <mingo elte.hu=""><br />[PATCH] Fix timeout overflow with jiffies<br />Prevent timeout overflow if timer ticks are behind jiffies <br />(due to high softirq load or due to dyntick), <br />by limiting the valid timeout range to MAX_LONG/2. </mingo></span><br />
<br />
So what can be done for <span style="font-family: "Courier New",Courier,monospace;"><b>msecs_to_jiffies()</b></span>?<br />
<br />
Instead of clamping negative values to MAX_JIFFY_OFFSET<br />
<span style="font-family: "Courier New",Courier,monospace;">502 /*</span><br />
<span style="font-family: "Courier New",Courier,monospace;">503 * Negative value, means infinite timeout:</span><br />
<span style="font-family: "Courier New",Courier,monospace;">504 */</span><br />
<span style="font-family: "Courier New",Courier,monospace;">505 if ((int)m < 0)</span><br />
<span style="font-family: "Courier New",Courier,monospace;">506 return MAX_JIFFY_OFFSET;</span><br />
<br />
Should values larger than (MAX_LONG/2) be clamped to MAX_JIFFY_OFFSET?<br />
<span style="font-family: "Courier New",Courier,monospace;">502 /*</span><br />
<span style="font-family: "Courier New",Courier,monospace;">503 * Prevent timeout overflow if timer ticks are behind jiffies </span><br />
<span style="font-family: "Courier New",Courier,monospace;">504 * by limiting the valid timeout range to MAX_LONG/2.</span><br />
<span style="font-family: "Courier New",Courier,monospace;">505 */</span><br />
<span style="font-family: "Courier New",Courier,monospace;">506 if (m > MAX_JIFFY_OFFSET)</span><br />
<span style="font-family: "Courier New",Courier,monospace;">507 return MAX_JIFFY_OFFSET;</span><br />
<br />
This update follows the spirit of the patch that introduced the current definition of MAX_JIFFY_OFFSET. Heck, even the comment is borrowed from it!<br />
<br />
To answer the original question, any invocation of <span style="font-family: "Courier New",Courier,monospace;">msleep(<i><b>N</b></i>)</span><br />
where <span style="font-family: "Courier New",Courier,monospace;">MAX_JIFFY_OFFSET < <i><b>N</b></i> < LONG_MAX</span><br />
technically sleeps for longer than "infinity" as defined in the context of msleep().<br />
<br />
For the technically inclined, a parting question -<br />
<br />
<blockquote class="tr_bq">
<b>Q</b>. With the current implementation in the Linux kernel, <br />
under what circumstances will <span style="font-family: "Courier New",Courier,monospace;"><b>msleep(10)</b></span> return sooner than 10ms?</blockquote>
Unknownnoreply@blogger.com0Bengaluru, Karnataka, India12.9715987 77.59456269999998312.4764182 76.949115699999979 13.4667792 78.240009699999987tag:blogger.com,1999:blog-5879050571098780501.post-42068375758037920572013-05-08T16:27:00.001+05:302013-05-13T13:12:47.538+05:30Sensors on Google Glass<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-n8FiKXo8t5M/UYou70T5b5I/AAAAAAAABKg/RnxMPEO6yJ0/s1600/google-glass.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="http://4.bp.blogspot.com/-n8FiKXo8t5M/UYou70T5b5I/AAAAAAAABKg/RnxMPEO6yJ0/s1600/google-glass.jpg" height="145" width="320" /></a></div>
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;">Recently Google has shared the <a href="https://android.googlesource.com/kernel/omap/+/glass-omap-xrr02" target="_blank">Linux Kernel source for the firmware running on the Google Glass</a>. Having grown tired of watching<span style="font-size: small;"> </span>online reviews about the monotonous "Glass this, glass that" voice commands, i was curious about the support for o<span style="font-size: small;">t</span>her forms of inpu<span style="font-size: small;">t.</span> (read gestures). A quick peek into the innards of the kernel source revealed quite a lo<span style="font-size: small;">t...</span></span></span><br />
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><span style="font-size: small;"> </span> </span></span><br />
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"></span></span></div>
<h2 style="text-align: justify;">
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;">FACT1. Google Glass runs on Texas Instruments OMAP4430.</span></span></h2>
<h4 style="text-align: justify;">
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"></span></span></h4>
<div style="text-align: justify;">
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;">Nothing revolutionary. A major win for TI (FWIW), considering that it has nonchalantly quit the mobile-SoC market citing a low RoI. This was already known as <a href="http://www.engadget.com/2013/04/26/google-glass-runs-on-omap-4430" target="_blank">some guy who actually had the Google <span style="font-size: small;">G</span>lass<span style="font-size: small;">,</span> ran adb and found out</a>.</span></span></div>
<div style="text-align: justify;">
<blockquote class="tr_bq">
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><b>Refer</b>: <a href="https://android.googlesource.com/kernel/omap/+/glass-omap-xrr02/arch/arm/configs/notle_defconfig" target="_blank">arch/arm/configs/notle_defconfig</a></span></span></blockquote>
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"> </span></span></div>
<div style="text-align: justify;">
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"></span></span></div>
<h2>
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><b>FACT2. Google Glass has a built-in </b></span></span><span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><b><span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><b> Accel, </b></span></span>Gyro & Compass.</b> </span></span></h2>
<div style="text-align: justify;">
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><b>Invensense MPU6050 </b>= 3axis gyro + 3 axis accel.<br /><b>Asahi Kasei AKM8975 </b>= 3axis geomagnetic sensor(compass).<br />Combining facts 1 and 2 we can see that the device spec for SoC and sensors perfectly matches the popular Samsung Galaxy S2 (variant <a href="http://www.gsmarena.com/samsung_i9100g_galaxy_s_ii-4327.php" target="_blank"><b>I9100G</b></a>).</span></span></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;">Rather than having independent ICs for both, the Google Glass uses MPU9150. <a href="http://www.invensense.com/mems/gyro/mpu9150.html" target="_blank"><b>Invensense MPU9150</b></a> is a single SiP which contains MPU6050 and AK8975 ICs within. This is fully hardware-compatible with existing MPU6050 board designs with the additional benefit of... (as Invensense quotes<span style="font-size: small;"> on its website</span>) "<i>...providing a simple upgrade path and making it easy to fit on space constrained boards.</i>" Perfect for Google Glass.</span></span><br />
<blockquote class="tr_bq">
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><b>Refer:</b> <a href="https://android.googlesource.com/kernel/omap/+/glass-omap-xrr02/arch/arm/mach-omap2/board-notle.c" target="_blank">arch/arm/mach<span style="font-size: small;">-</span>omap2/board<span style="font-size: small;"><span style="font-size: small;">-</span>notle.c</span></a><span style="font-size: small;"> line:</span>1710</span></span></blockquote>
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"></span></span></div>
<div style="text-align: justify;">
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><br /></span></span></div>
<div style="text-align: justify;">
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"></span></span></div>
<h2>
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><b>FACT3. Google Glass has a "glasshub"</b></span></span></h2>
<h2>
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"></span></span></h2>
<div style="text-align: justify;">
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;">I stumbled upon this by accident as i was searching for the sensor drivers. The glasshub appears to be a external micro<span style="font-size: small;">-</span>controller that communicates with OMAP4430 over I2C. This is the hardware that supports the "wink" command. Strangely enough, it supports upto 20winks! Looks like someone didn't learn their lesson with triple and quadruple mouse-clicks designs. On the other hand, this will be most essential when someone attempts to write a Google Glass app to detect seizures. Forward thinking as always, Google.<br /><br />The glasshub also reports IR data and proximity (not sure about the underlying hardware though).</span></span></div>
<blockquote class="tr_bq">
<div style="text-align: justify;">
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><b>Refer: </b><a href="https://android.googlesource.com/kernel/omap/+/glass-omap-xrr02/arch/arm/mach-omap2/board-notle.c" target="_blank">arch/arm/mach<span style="font-size: small;">-</span>omap2/board<span style="font-size: small;"><span style="font-size: small;">-</span>notle.c</span></a><span style="font-size: small;"> line:</span>1<span style="font-size: small;">843 </span></span></span><br />
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><span style="font-size: small;"> </span></span></span><a href="https://android.googlesource.com/kernel/omap/+/glass-omap-xrr02/drivers/input/misc/glasshub.c" target="_blank">drivers/input/mis/glasshub.c</a> </span></span></div>
</blockquote>
<div style="text-align: justify;">
</div>
<h2>
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><b> </b></span></span></h2>
<h2>
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><b>FACT4. Google Glass has a "Proximity" sensor.</b></span></span></h2>
<h2>
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"></span></span></h2>
<div style="text-align: justify;">
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;">Not to be confused with the "glasshub" there is another independent module<span style="font-size: small;">,</span> the <b>LiteON LTR-506ALS</b>. A very good sign, this IC is extremely customisable when it comes thresholds/IR-pulse-freq/pulse-count/poll-rate. Maybe, just maybe, we c<span style="font-size: small;">oul</span>d hack the whole setup <span style="font-size: small;">into</span> a rudimentary IR remote. While being used primarily for ambient light sensing, it also supports proximity sensing. <span style="font-size: small;">T</span>his means that we can have the Google Glass detecting our finger/hand swipes in front of our face. Quite the most exciting tech of the lot as it will provide the illusion of being able to actually handle the projected images.</span></span><br />
<blockquote class="tr_bq">
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><b>Refer:</b> <a href="https://android.googlesource.com/kernel/omap/+/glass-omap-xrr02/arch/arm/mach-omap2/board-notle.c" target="_blank">arch/arm/mach<span style="font-size: small;">-</span>omap2/board<span style="font-size: small;"><span style="font-size: small;">-</span>notle.c</span></a><span style="font-size: small;"> line:</span>17<span style="font-size: small;">27</span> </span></span><br />
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><span style="font-size: small;"> </span></span></span><a href="https://android.googlesource.com/kernel/omap/+/glass-omap-xrr02/drivers/input/misc/ltr506als.c" target="_blank">drivers/input/misc/ltr506als.c</a></span></span></blockquote>
</div>
<div style="text-align: justify;">
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"></span></span></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><br /><span style="font-size: small;"></span></span></span>
<br />
<div style="text-align: center;">
<span style="font-family: Verdana,sans-serif;">Overall quite a good amount of<br /> "sensory" tech inside for me to play with.<br />Me so excited. ;-) ;-) wink wink</span></div>
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"></span></span></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-2ZoET5eGmWg/UYoqtCQ67zI/AAAAAAAABKQ/pkE5pn6AK9E/s1600/PiB.jpeg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://4.bp.blogspot.com/-2ZoET5eGmWg/UYoqtCQ67zI/AAAAAAAABKQ/pkE5pn6AK9E/s1600/PiB.jpeg" height="288" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-size: small;"><span style="font-family: Verdana,sans-serif;">Hey Google, Can i haz a Google Glass?</span></span></td></tr>
</tbody></table>
<div style="text-align: justify;">
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"></span></span></span></span></div>
<div style="text-align: center;">
<span style="font-family: Verdana,sans-serif;"><br /></span>
</div>
Unknownnoreply@blogger.com10Puttappa layout, New Thippasandra, Bangalore, Karnataka, India12.971433099459315 77.65763282775878912.970466099459316 77.656372327758788 12.972400099459314 77.65889332775879tag:blogger.com,1999:blog-5879050571098780501.post-39251645789730946582013-02-19T11:20:00.000+05:302014-08-06T20:56:38.645+05:30Internet Radio : Part1 - Shoutcast Protocol<div style="text-align: justify;">
<blockquote class="tr_bq">
NOTE: If you spent the last decade 100miles below the surface of the
earth studying the growth of algae under an antarctic glacier, then you
will be surprised to learn that we can now listen to radio over the
internet. Just like tuning-in to particular frequencies for particular
stations in the olden days; now we can "tune-in" to specific radio
stations over the internet. Without further ado, lets jump-in into the
world of "Internet Radio".</blockquote>
</div>
<div style="text-align: justify;">
<br />
Most apps claiming to support Internet Radio, in fact support a industry standard - <b><a href="http://en.wikipedia.org/wiki/SHOUTcast" target="_blank">Shoutcast</a></b>. It was a protocol devised by nullsoft in the 90's and first implemented in their popular player <b><a href="http://en.wikipedia.org/wiki/Winamp" target="_blank">Winamp</a></b>(stop
reading if you haven't heard of Winamp!). Inspite of being a
proprietary protocol with not much documentation to go with, Shoutcast
has become the de-facto industry standard for streaming audio. This is
mainly due to its simplicity and similarity with the existing hyper-text
transfer protocol (dear old http! wink wink). <b><a href="http://en.wikipedia.org/wiki/Icecast" target="_blank">Icecast</a></b> is a similar open-source implementation compatible with Shoutcast.</div>
<div style="text-align: justify;">
<br /></div>
<h2 style="text-align: justify;">
Initial handshake between Shoutcast client-server </h2>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-qlzhouvRNpA/USIWJEiWbyI/AAAAAAAABAo/TSb62M1uApM/s1600/shoutcast-initial.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://3.bp.blogspot.com/-qlzhouvRNpA/USIWJEiWbyI/AAAAAAAABAo/TSb62M1uApM/s1600/shoutcast-initial.jpg" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">High-level overview of Internet-radio over Shoutcast. </td></tr>
</tbody></table>
<div style="text-align: justify;">
<h4>
[STEP1] Station Listing</h4>
The client app connects to a station listing/aggregator on the internet and obtains a list of stations alongwith their details like genres, language, now-playing, bitrate among other things.<br />
<br />
<h4>
</h4>
<h4>
[STEP2] Station Lookup</h4>
The user can then select one of the stations as desired. Then the client obtains the ip-address(& port) of the server running that particular station from the station-listing/aggregator. Networking enthusiasts will notice that this step is exactly like a DNS lookup i.e. the client obtains the network address for a particular station name; the station-listing/aggregator acting like a DNS-server for Radio stations. Also note that sometimes the station-listing will provide only a domain-name and then an additional actual DNS lookup is needed to obtain the ip-address of the streaming server. Popular station-listing/aggregator sites like <b><a href="http://dir.xiph.org/" target="_blank">Xiph</a></b>, <b><a href="http://shoutcast.com/">Shoutcast.com</a></b> and <b><a href="http://vtuner.com/setupapp/guide/asp/BrowseStations/startpage.asp" target="_blank">vTuner</a></b> provide huge web-friendly lists of live radio stations.<br />
<br />
<h4>
[STEP3|4] Station Connection</h4>
1. The client attempts to connect to the server using the ip-address(and port) obtained during station lookup. <br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://lh6.googleusercontent.com/-og3mEkbCP3c/USIsqstdvpI/AAAAAAAABBM/UUxRs2a6qd0/s1600/shoutcast-connect-request.png" target="_blank"><img border="0" src="http://1.bp.blogspot.com/-og3mEkbCP3c/USIsqstdvpI/AAAAAAAABBM/UUxRs2a6qd0/s400/shoutcast-connect-request.png" height="210" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://lh6.googleusercontent.com/-og3mEkbCP3c/USIsqstdvpI/AAAAAAAABBM/UUxRs2a6qd0/s1600/shoutcast-connect-request.png" target="_blank">Connection request from shoutcast client (click to enlarge)</a></td></tr>
</tbody></table>
<br />
2. The server responds with "ICY 200 OK" (a custom <a href="http://en.wikipedia.org/wiki/List_of_HTTP_status_codes#2xx_Success" target="_blank">200 OK success code</a>)...<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-OgXyHSSH8SQ/USIssBf3TmI/AAAAAAAABBU/AdHnv3ctvlI/s1600/ICY200OK.png" imageanchor="1" style="margin-left: auto; margin-right: auto;" target="_blank"><img border="0" src="http://3.bp.blogspot.com/-OgXyHSSH8SQ/USIssBf3TmI/AAAAAAAABBU/AdHnv3ctvlI/s400/ICY200OK.png" height="208" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="http://3.bp.blogspot.com/-OgXyHSSH8SQ/USIssBf3TmI/AAAAAAAABBU/AdHnv3ctvlI/s1600/ICY200OK.png" target="_blank">ICY 200 OK reply from shoutcast server (click to enlarge)</a></td></tr>
</tbody></table>
<br />
3. ...and the stream header...<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://lh4.googleusercontent.com/-M1IqNOvFT10/USIs1Dtlt-I/AAAAAAAABBc/KV1a4DPp71U/s1600/shoutcast-header.png" imageanchor="1" style="margin-left: auto; margin-right: auto;" target="_blank"><img border="0" src="http://4.bp.blogspot.com/-M1IqNOvFT10/USIs1Dtlt-I/AAAAAAAABBc/KV1a4DPp71U/s400/shoutcast-header.png" height="210" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://lh4.googleusercontent.com/-M1IqNOvFT10/USIs1Dtlt-I/AAAAAAAABBc/KV1a4DPp71U/s1600/shoutcast-header.png" target="_blank">Shoutcast stream header (click to enlarge)</a></td></tr>
</tbody></table>
<br />
4. ..and finally the server starts sending encoded audio in a continuous stream of packets(which the client app can decode and playback) until the client
disconnects(stops ACK-ing and signals a disconnect). <br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://lh5.googleusercontent.com/-dTB_v67HT1o/USIs2smMDmI/AAAAAAAABBk/i19r2PC7sg0/s1600/shoutcast-data.png" imageanchor="1" style="margin-left: auto; margin-right: auto;" target="_blank"><img border="0" src="http://1.bp.blogspot.com/-dTB_v67HT1o/USIs2smMDmI/AAAAAAAABBk/i19r2PC7sg0/s400/shoutcast-data.png" height="210" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://lh5.googleusercontent.com/-dTB_v67HT1o/USIs2smMDmI/AAAAAAAABBk/i19r2PC7sg0/s1600/shoutcast-data.png" target="_blank">Encoded audio data stream (click to enlarge)</a></td></tr>
</tbody></table>
<blockquote class="tr_bq">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://docs.google.com/file/d/0B86sbaayeRPSXzBhUVkwTXRJSFk/edit?usp=sharing" target="_blank"><b style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://upload.wikimedia.org/wikipedia/commons/d/db/Wireshark_Icon.png" height="32" width="32" /></b></a></div>
<a href="https://docs.google.com/file/d/0B86sbaayeRPSXzBhUVkwTXRJSFk/edit?usp=sharing" target="_blank">Download the entire WireShark capture of packets</a> exchanged by the shoutcast client and server during initial station connection.<b> </b></blockquote>
<br />
The above steps are similar to what a browser does when it connects to a website (and hence in-browser streaming audio playback of shoutcast streams IS possible).<br />
<br />
Shoutcast has <a href="http://forums.radiotoolbox.com/viewtopic.php?t=74" target="_blank">subtle differences</a> over http during the <b>station connection</b> step above. Shoutcast supports "<b>Icy-MetaData</b>" - an additonal field in the request header. When set, its a request to the shoutcast server to embed metadata about the stream at periodic intervals(once every "<b>icy-metaint</b>" bytes) in the encoded audio stream itself. The value of "<b>icy-metaint</b>" is decided by the shoutcast server configuration and is sent to the client as part of the initial reply.</div>
<div style="text-align: justify;">
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-BJMe7DT59P0/USIblA2sp_I/AAAAAAAABA4/YHLZ82QMD6g/s1600/shoutcast-metadata.jpg" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://4.bp.blogspot.com/-BJMe7DT59P0/USIblA2sp_I/AAAAAAAABA4/YHLZ82QMD6g/s1600/shoutcast-metadata.jpg" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Shoutcast stream format when ICY:MetaData is set to 1</td><td class="tr-caption" style="text-align: center;"><br /></td></tr>
</tbody></table>
<div style="text-align: justify;">
<br />
This poses a slight complication during playback. If the received audio stream is directly queued for playback, then the embedded metadata appears as periodic glitches. Following is one such sample recording.
This audio clip was retrieved from a radio stream whose icy:metaint =
32768; i.e. the metadata is embedded in the audio stream once every
32KBytes. Stream bit-rate is 4KBps. <b>So during playback a glitch is
present once every 32KB/4KB = 8seconds (0:08s, 0:16s, 0:24s, 0:32s,...).</b><br />
<br />
<iframe frameborder="no" height="166" scrolling="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F79796804&color=ff7f00&auto_play=false&show_artwork=false" width="100%"></iframe>
<br />
<br />
<strike>To view/analyse the stream data in a hex editor, <a href="https://soundcloud.com/thecodeartist/shoutcast-metadata" target="_blank">download the actual clip</a> and check out the following offsets </strike><b> </b><br />
<blockquote class="tr_bq">
<b>Update: </b>Unfortunately the service i was using to host the audio clip has lost it and i was foolish enough to trust them and have no local backups. :(<br />
<div style="text-align: center;">
R.I.P<br />
Shoutcast-Metadata.mp3 </div>
<div style="text-align: center;">
2013-2014<br />
Here lies a song, cut short in its prime...</div>
</blockquote>
<br />
[0:08s] 0x0815A - 0x0817A count N = 2, meta = 1+ (16 x 2) = 33(0x21h)bytes<br />
[0:16s] 0x1017B - 0x1017B count N = 0, meta = 1byte<br />
[0:24s] 0x1817C - 0x1817C count N = 0, meta = 1byte <br />
[0:32s] 0x2017D - 0x2017D count N = 0, meta = 1byte<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-GYwRA3LCmXY/USMMRODF3JI/AAAAAAAABB4/iUsA-9lQpIY/s1600/stream-metadata-hex.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://3.bp.blogspot.com/-GYwRA3LCmXY/USMMRODF3JI/AAAAAAAABB4/iUsA-9lQpIY/s480/stream-metadata-hex.jpg" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Embedded metadata from 0x0815A to 0x0817a.<br />
Note the first byte is 02 i.e. metadata is 2x16=32(0x20h)bytes following it.</td></tr>
</tbody></table>
<br />
Also note that the first 345(0x159h)bytes of the clip are the reply header of the
stream(plain-text in ASCII) sent by the shoutcast server. Technically
these are NOT part of the audio stream as well.<br />
<br />
NOTE: If you simply want to obtain the audio stream (no embedded metadata) then set the "<b>Icy-MetaData" field in the request header to 0 </b>or simply do NOT pass it as part of the initial request header.<br />
<br />
Finally here is a small bit of code that implements all that we have learnt so far - a simple shoutcast client in a few lines of C, that connects to any shoutcast server and logs the audio stream data to stdout. It uses the curl library to initiate connection requests to the shoutcast server.<br />
<br />
<script src="https://gist.github.com/TheCodeArtist/2f1b9fa68197e39ca9bc.js"></script><a href="https://gist.github.com/TheCodeArtist/2f1b9fa68197e39ca9bc">https://gist.github.com/TheCodeArtist/2f1b9fa68197e39ca9bc</a><br />
<span style="font-size: x-small;"><span style="font-size: small;">Stripping off the comments and the clean-up code following line:50, it comes down to 13 lines of C code. Pretttty neat eh?...</span></span><br />
<br />
<blockquote class="tr_bq">
Usage:<br />
$> <b>sudo apt-get install libcurl4-gnutls-dev</b><br />
$> <b>gcc simple.c -o simple -l libcurl</b><br />
$> <b>./simple </b><<i>shoutcast-server-ip-addr:port</i>><b> > </b><<i>test-file</i>> </blockquote>
<br />
<span style="font-size: small;">After running the above commands, the <i><b><span style="font-size: small;"><</span>test-file<span style="font-size: small;">></span></b></i></span> <test .file="">will contain the audio stream of that particular internet radio station. The <test-file> can be played back in any player that supports decoding the stream format(AAC, MP3, OGG etc. depending on the radio station) Make sure to comment out line 38 in simple.c to have a glitch-free(no embedded metadata) audio stream.</test-file></test><br />
<br />
<blockquote class="tr_bq">
This concludes part 1 of the series on how internet radio works. In part2 we will analyse the challenges and issues faced during de-packetising, parsing and queuing the audio stream buffers for local playback. <a href="http://feeds.feedburner.com/TheCodeArtist" target="_blank"><b>Stay tuned for updates</b></a>.</blockquote>
<br />
<br /></div>
Unknownnoreply@blogger.com2Bangalore, Karnataka, India12.9715987 77.59456269999998312.4764197 76.949115699999979 13.4667777 78.240009699999987tag:blogger.com,1999:blog-5879050571098780501.post-49936499286805101702012-08-28T19:15:00.003+05:302014-02-18T01:30:58.135+05:30HDD, FS, O_SYNC : Throughput vs. Integrity<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="font-family: Verdana,sans-serif; text-align: justify;">
Today we will spend some time over filesystems, block-devices, throughput and data-integrity. But first, a few "MYTHBUSTER" statements.</div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<blockquote class="tr_bq">
#1: Even the fastest HDD today can do ONLY 650KBps natively.</blockquote>
</div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<br /></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<blockquote class="tr_bq">
#2: O_SYNC on a filesystem does NOT guarantee a write to the HDD.</blockquote>
</div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<br /></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<blockquote class="tr_bq">
#3: Raw-I/O over BLOCK devices DOES guarantee data-integrity.</blockquote>
</div>
<div style="text-align: justify;">
<br />
<span style="font-family: Verdana,sans-serif;"><b>Hard to believe, right? Lets analyse these statements one by one...</b></span><br />
<br />
<span style="font-family: Verdana,sans-serif;">The fastest Hard-disk drives today run at 10,000RPM (compared to regular
ones at 5400,7200). Also the faster HDDs have transitioned to a 4096
byte internal block-size (compared to 512 bytes on regular ones).</span> </div>
<div style="text-align: justify;">
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-ksvQJ3xe-n8/UDzF09TQOtI/AAAAAAAAA9E/Sj6uFQy-3fw/s1600/hdd-params.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://2.bp.blogspot.com/-ksvQJ3xe-n8/UDzF09TQOtI/AAAAAAAAA9E/Sj6uFQy-3fw/s1600/hdd-params.jpg" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-size: small;">Details of <b><a href="http://en.wikipedia.org/wiki/Hard_disk_drive#Components" target="_blank">HDD components</a></b></span></td></tr>
</tbody></table>
<div style="text-align: justify;">
</div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
To read/write one particular sector from/to the HDD, the head needs to be first aligned radially in the proper position. Next one waits as the rotating disk platter positions the desired sector under the disk-head. Now one is able read/write to the sector.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-bqwnoZXFodA/UDzGukxpIyI/AAAAAAAAA9M/gasjxs9XYiM/s1600/disk-head-platter.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-bqwnoZXFodA/UDzGukxpIyI/AAAAAAAAA9M/gasjxs9XYiM/s1600/disk-head-platter.jpg" /></a></div>
</div>
<div class="separator" style="clear: both; text-align: justify;">
</div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<div style="text-align: center;">
Unit of data on HDD = 1 sector =4096 bytes</div>
<div style="text-align: center;">
MAX RPM = 10,000 = 10,000/60 = 166.667 rotations/second</div>
<div style="text-align: center;">
Seek-time = 1/166.667 = 0.006s</div>
<div style="text-align: center;">
Throughput = 4096/0.006 = 682666.667 ~ <u><b>650KBps</b></u></div>
<br />
The above calculation assumes the worst possible values for both the I/O-size(1 sector) and seek-time(1 entire rotation). This condition though is quite easily seen in real life scenarios like database applications which use the HDD as a raw block-device.<br />
<br />
Better speeds in the range of 20-50MBps are commonly obtained by a combination of several strategies like:</div>
<ul style="font-family: Verdana,sans-serif; text-align: justify;">
<li>Multi-block I/O.</li>
<li>Native Command Queueing.</li>
<li>RAID stripping. </li>
</ul>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
Now lets consider a regular I/O request at the HDD level:<br />
<h2>
HDD Read:</h2>
</div>
<ol style="font-family: Verdana,sans-serif; text-align: justify;">
<li>The kernel raises a disk read request to the HDD.The HDD has a small amount of disk-cache (RAM) which it checks to see if the requested data exists.</li>
<li>If NOT found in the disk-cache then the disk-head is moved to the data location on the platter.</li>
<li>The data is read into disk-cache and a copy is returned to the kernel.</li>
</ol>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<h2>
HDD Write:</h2>
</div>
<ol style="font-family: Verdana,sans-serif; text-align: justify;">
<li>The kernel raises a disk write request to the HDD.</li>
<li>The HDD has a small amount of cache (RAM) where the data to be written is placed.</li>
<li style="font-family: Verdana,sans-serif;">An on-board disk controller is in-charge of the saving the contents of the cache to the platter. It operates on the following set of rules:</li>
</ol>
<ul style="font-family: Verdana,sans-serif; text-align: justify;">
<li>[Rule1] Write cache to platter as soon as possible.</li>
<li>[Rule2] Re-order write operations in sequence of platter location.</li>
<li>[Rule3] Bunch several random writes together into one sequence.</li>
</ul>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-Vlm3TrD1nvQ/UDzJDy8axDI/AAAAAAAAA9U/QByOAxFAC_s/s1600/T-800-disk-controller.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://1.bp.blogspot.com/-Vlm3TrD1nvQ/UDzJDy8axDI/AAAAAAAAA9U/QByOAxFAC_s/s1600/T-800-disk-controller.jpg" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-size: small;">"Hasta la vista, baby!"<br />HUD of the disk-IO firmware running on a <b><a href="http://www.youtube.com/watch?v=jp5KTSaFhY0" rel="nofollow" target="_blank">T-800 Terminator</a></b> ;-)<b><a href="http://www.youtube.com/watch?v=9MeaaCwBW28" target="_blank"><br /></a></b></span></td></tr>
</tbody></table>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
[Rule1] <u>minimises data loss</u>. As cache is volatile i.e. any power-outage will mean that data (in the HDD-cache), which is NOT yet written to the HDD-platter, is lost.<br />
<br />
[Rule2] <u>optimises the throughput</u>. As serialising access reduces the time spent in seeking by the read-write head.<br />
<br />
[Rule3] <u>reduces power consumption, disk-wear</u> by allowing the disk to be stopped from constantly spinning all the time. Only when the cache is filled to a certain limit the disk motor is powered on and the cache is flushed to the platter, following which the motor is powered-down again until the next cache flush.<br />
<br />
Its obvious that [1] [2] [3] are counter-productive and the right balance needs to be struck between the three to have data-integrity, high-throughput, low-power-consumption & longer disk-life. Several complex algorithms have been devised to handle this in modern-day HDD controllers. <br />
<br />
The problem though is that by default performance is sacrificed in favour of the other two. This is just a "default" setting though and the beauty of the Linux-Kernel being open-source is that one is free to stray from the "default" setting.<br />
<br />
<blockquote class="tr_bq">
If you are doing <u>mostly sequential raw block-I/O</u> on a SATA HDD, you would be a prime candidate for this <b><a href="https://gist.github.com/93dddcd6a21dc81414ba" target="_blank">patch</a></b>, which effectively moves the operation-point of the HDD closer to high-performance region in the map.</blockquote>
</div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-6fqn622U-oI/UDzJmRM3aGI/AAAAAAAAA9c/l7W0gcoMoJY/s1600/perf-power-data.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-6fqn622U-oI/UDzJmRM3aGI/AAAAAAAAA9c/l7W0gcoMoJY/s1600/perf-power-data.jpg" /></a></div>
<br />
Moving on, we now focus on how the use of O_SYNC affects I/O on filesystems as well as the raw block device.</div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-GAfGxWre7UM/UDzKLJDqziI/AAAAAAAAA9k/4-1Wiy-i9N4/s1600/data-barriers-hdd-write.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-GAfGxWre7UM/UDzKLJDqziI/AAAAAAAAA9k/4-1Wiy-i9N4/s1600/data-barriers-hdd-write.jpg" /></a></div>
<div class="separator" style="clear: both; text-align: justify;">
</div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<br />
<h2>
Regular write on a regular filesystem</h2>
</div>
<div style="text-align: justify;">
<blockquote class="tr_bq" style="font-family: "Courier New",Courier,monospace;">
<b>fd = open("/media/mount1/file");</b></blockquote>
</div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
Consider the first case where one does a regular write(NO O_SYNC flag) on a regular filesystem on a HDD. The data is copied from the APP to the FS i.e. userspace-app(RAM) into the kernel filesystem page-cache(RAM) and control returns. During this, one does NOT cross any data-barriers and hence data is NOT guaranteed to be written to the HDD. This makes the entire process of a regular write on an fs extremely fast.<br />
<br /></div>
<h2>
Synchronous write on a regular filesystem</h2>
<div style="text-align: justify;">
<blockquote class="tr_bq">
<b style="font-family: "Courier New",Courier,monospace;">fd = open("/media/mount1/file", O_SYNC);</b></blockquote>
</div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
The second case illustrated above depicts a synchronous write (with O_SYNC flag) on a regular filesystem on a HDD. The man page for open() call contains the following notes:</div>
<div style="text-align: justify;">
<blockquote class="tr_bq" style="font-family: Verdana,sans-serif;">
O_SYNC The file is opened for synchronous I/O. Any writes on the resulting file descriptor will block the calling process until the data has been physically written to the underlying hardware.</blockquote>
</div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<br />
Although using O_SYNC looks like a sureshot guarantee that data is indeed written to disk, there lies a catch in the implementation. Most HDDs contain a on-board cache on the HDD itself. A write command to the disk transfers the data from the kernel filesystem page-cache(RAM) to the HDD-cache(not the actual mechanical platter of the disk.) This process is limited by the bus(SATA, IDE etc) which is faster than the actual mechanical platter. When a HDD receives a write command, it copies the data into its internal HDD-cache and returns immediately. The HDD's internal firmware later transfers the data to the disk-platter according to its "3 rules" as discussed previously. Thus a write to HDD does NOT necessarily imply a write to the disk platter. Hence even synchronous writes on a filsystem do NOT imply 100% data integrity.<br />
<br />
Also a data-barrier exists along this path in the filesystem layer where metadata(inode,superblock) info is stored. This will help in identifying any data integrity on future access. Note that maintaining/updating inode,superblocks does NOT guarantee that the data is written to disk. Rather it makes the last sequence of writes atomic (i.e. all the writes get committed to disk or none). The inode,superblock info serves as a kind of checksum as they are updated accordingly following the atomic write operation. All this processing means that throughput incurs a slight penalty in this case.</div>
<div style="text-align: justify;">
<br />
<h2>
Synchronous write on a block device</h2>
<blockquote class="tr_bq">
<b style="font-family: "Courier New",Courier,monospace;">fd = open("/dev/sda", O_SYNC);</b></blockquote>
</div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
The third case illustrated above is a synchronous write to the HDD directly via its block-device(eg. /dev/sda). In this case there is NO data-barrier in the filesystem. The O_SYNC is implemented by using the data-barrier present in the HDD i.e. flushing the disk-cache explicity to ensure that all the data is indeed transferred to the disk-platter before returning. This incurs the maximum penalty and hence the throughput is the slowest of all 3 scenarios above.<br />
<br />
<h2>
Salient observations:</h2>
</div>
<ol style="font-family: Verdana,sans-serif; text-align: justify;">
<li>A data barrier is a module/function across which data integrity is guaranteed i.e if the function is called and it returns successfully, then the data is completely written to non-volatile memory(HDD, in this case). A data barrier introduces an order of magnitude change in:</li>
<ul>
<li>Access-time (++)</li>
<li>Throughput (--)</li>
</ul>
<li>A data barrier in the lower layers incurs a larger penalty than one in the upper layers. (Penalty : App < FS < Disk.)</li>
<li>O_SYNC on HDD via filesystems does NOT guarantee a successful write to non-volatile disk-platter.</li>
<li>O_SYNC on HDD via block-device directly guarantees data-integrity but offers very low throughput.</li>
<li>The HDD data-barrier(FLUSH_CACHE) is NOT utilised when using regular filesystems to access the HDD.</li>
<li><b><a href="https://gist.github.com/93dddcd6a21dc81414ba" target="_blank">Disabling HDD data-barrier</a></b> and raw DIRECT-I/O via the block device provides maximum throughput to a HDD.</li>
</ol>
<div style="text-align: justify;">
<blockquote class="tr_bq">
<b style="font-family: "Courier New",Courier,monospace;">fd = open("/dev/sda", O_SYNC|O_DIRECT);</b></blockquote>
<div style="font-family: Verdana,sans-serif;">
<br /></div>
<div style="font-family: Verdana,sans-serif;">
Further reading : <b><a href="http://thecodeartist.blogspot.com/2012/06/improving-hdd-performance-linux.html" target="_blank">5 ways to improve HDD performance on Linux</a></b></div>
</div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
</div>
</div>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-5879050571098780501.post-74107735481815870692012-06-15T13:16:00.001+05:302012-09-16T13:05:36.038+05:30SATA hotplug : Add/Remove sata HDD in a jiffy<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
Its not common knowledge that SATA HDDs are capable of being hot-plugged into almost any modern PC. However using them in a plug-n-play manner like portable external USB-HDDs is still not common.</div>
<br />
Here is how to use your SATA HDD like an portable HDD.<br />
<br />
<u>Tested on:</u><br />
- Ubuntu 11.10<br />
- DELL Optiplex 380<br />
- Seagate Barracuda 1TB (SATA).<br />
<br />
<h2>
1. Connect the SATA HDD to host PC. (sata-bus + power) </h2>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" src="http://3.bp.blogspot.com/-pW_Norp8VCI/T9roDCtfy1I/AAAAAAAAA2k/rNvlFlgxmMg/s1600/sata-hdd.jpg" /></div>
<br />
<h2>
2. Scan for new devices on SCSI host</h2>
<blockquote class="tr_bq">
sudo echo "- - -" > /sys/class/scsi_host/host<b>N</b>/scan</blockquote>
where <b>N</b> is the host port number on your host PC to which you have plugged-in the SATA HDD. Usually N=1, assuming the primary HDD on host PC is connected on SATA0.<br />
<br />
"- - -" stands for wildcards in place of the<br />
channel number, SCSI target ID, and LUN. <a href="http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Online_Storage_Reconfiguration_Guide/adding_storage-device-or-path.html" target="_blank"><b>More Info</b></a><br />
<br />
<h2>
3. Mount the newly detected device locally </h2>
<blockquote class="tr_bq">
sudo mount /dev/sd<b>X</b> /media/temphdd</blockquote>
where <b>X</b> is a/b/c/d etc. Usually X=b, assuming the primary HDD on host PC is enumerated as sda an there are no other block devices.<br />
<br />
<h2>
4. Copy all your data to/from the HDD present at /media/tempHDD. </h2>
<br />
<h2>
5. Once finished, unmount the device</h2>
<blockquote class="tr_bq">
sudo umount /media/temphdd</blockquote>
<br />
<h2>
6. Powering down the SATA HDD</h2>
<blockquote class="tr_bq">
sudo echo 1 > /sys/block/sd<b>X</b>/device/delete</blockquote>
Ensure that you refer to the proper device (sdb, sdc etc.) as above in step 3.<br />
<br />
<h2>
7. Disconnect the SATA HDD from host PC.</h2>
Thats it! Thats how one can use the hotplug feature of SATA HDDs to efectively use them as portable external HDDs.</div>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-5879050571098780501.post-11331354259678873612012-06-13T15:10:00.001+05:302014-02-18T20:52:57.397+05:305 ways to improve HDD speed on Linux<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="text-align: center;">
LINUX <b>LINUX</b> LINUX <b>LINUX</b></div>
<div style="text-align: center;">
<b>LINUX</b> LINUX <b>LINUX</b> LINUX</div>
<br />
<div style="text-align: center;">
<span style="font-size: x-small;">(If you still think this post is about making windows load faster, then press ALT+F4 to continue)</span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-AipPFQfn-C8/T9rqKsY4lBI/AAAAAAAAA2s/FBKzaCn_qqw/s1600/Y-U-no-go-faster-hdd.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://4.bp.blogspot.com/-AipPFQfn-C8/T9rqKsY4lBI/AAAAAAAAA2s/FBKzaCn_qqw/s1600/Y-U-no-go-faster-hdd.jpg" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Our dear Mr. Client</td></tr>
</tbody></table>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div style="text-align: justify;">
<div style="text-align: justify;">
Its a fine sunny sunday morning. Due tomorrow, is your presentation to a client on improving disk-I/O. You pull yourself up by your boots and manage to climb out of bed and onto your favorite chair...</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-OE-880QqLoo/T9sJmEbsLqI/AAAAAAAAA3I/pknLzIaFO4c/s1600/blurry-window.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://1.bp.blogspot.com/-OE-880QqLoo/T9sJmEbsLqI/AAAAAAAAA3I/pknLzIaFO4c/s1600/blurry-window.jpg" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">I think you forgot to wear your glasses...</td></tr>
</tbody></table>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" src="http://4.bp.blogspot.com/-W-_DuM3mnbQ/T9sJsF-3IQI/AAAAAAAAA3Q/jNd3_n4WbXw/s1600/clear-window.jpg" /></div>
<br />
<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
Aahh thats better... You jump into the couch and turn on your laptop and launch <u>(your favorite presentation app here)</u>. As you sip your morning coffee and wait for the app to load, you look out of the window and wonder what it could be doing.</div>
<div style="text-align: justify;">
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-ZCvvVTTP9Pg/T9sJy0mC5oI/AAAAAAAAA3Y/Irk17AweJWU/s1600/disk-io-simple.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://3.bp.blogspot.com/-ZCvvVTTP9Pg/T9sJy0mC5oI/AAAAAAAAA3Y/Irk17AweJWU/s1600/disk-io-simple.jpg" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Looks simple, right? Then why is it taking so long?</td></tr>
</tbody></table>
</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" src="http://4.bp.blogspot.com/-U3stpdaHv3g/T9sJ8ExdvFI/AAAAAAAAA3g/b_66LGqcUG0/s1600/we-need-to-go-deeper.jpg" /></div>
<br />
<div style="text-align: justify;">
If you would be so kind enough to wipe your rose-coloured glasses clean, you would see that this is what is ACTUALLY happening</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" src="http://2.bp.blogspot.com/-I8SzXiayCrU/T9sKCKVNqVI/AAAAAAAAA3o/YGngspgi9-U/s1600/disk-io-detailed.jpg" /></div>
<br />
<div style="text-align: justify;">
0. The app (running in RAM) decides that it wants to play spin-the-wheel with your hard-disk.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
1. It initiates a disk-I/O request to read some data from the HDD to RAM(userspace).</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
2. The kernel does a quick check in its page-cache(again in RAM) to see if it has this data from any earlier request. Since you just switched-on your computer,... </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
3. ...the kernel did NOT find the requested data in the page-cache. "Sigh!" it says and starts its bike and begins it journey all the way to HDD-land. On its way, the kernel decides to call-up its old friend "HDD-cache" and tells him that he will be arriving shortly to collect a package of data from HDD-land. HDD-cache, the good-friend as always, tells the kernel not to worry and that everything will be ready by the time he arrives in HDD-land.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
4. HDD-cache starts spinning the HDD-disk...</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
5. ...and locates and collects the data.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
6. The kernel reaches HDD-land and picks-up the data package and starts back.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
7. Once back home, it saves a copy of the package in its cache in case the app asks for it again. (Poor kernel has NO way of knowing that the app has no such plans).</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
8. The kernel gives the package of data to the app...</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
9. ...which promptly stores it in RAM(userspace).</div>
<br />
<div style="text-align: justify;">
Do keep in mind that this is how it works in case of extremely disciplined, well-behaved apps. At this point misbehaving apps tend to go - </div>
<br />
<blockquote class="tr_bq">
"Yo kernel, ACTUALLY, i didn't allocate any RAM, i wanted to just see if the file existed. Now that i know it does, can you please send me this other file from some other corner of HDD-land."</blockquote>
...and the story continues...<br />
<br />
Time to go refill your coffee-cup. Go go go...<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-_tkdpkqDJPE/T9sKJJO3zSI/AAAAAAAAA3w/fksNDwUYlEM/s1600/coffee-n-donuts.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://2.bp.blogspot.com/-_tkdpkqDJPE/T9sKJJO3zSI/AAAAAAAAA3w/fksNDwUYlEM/s1600/coffee-n-donuts.jpg" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Hmmm... you are back with some donuts too. Nice caching!</td></tr>
</tbody></table>
<br />
<div style="text-align: justify;">
So as you sit there having coffee and donuts, you wonder how does one really improve the disk-I/O performance. Improving performance can mean different things to different people:</div>
<br />
<ol style="text-align: left;">
<li>Apps should NOT slow down waiting for data from disk.</li>
<li>One disk-I/O-heavy app should NOT slow down another app's disk-I/O.</li>
<li>Heavy disk-I/O should NOT cause increased cpu-usage.</li>
<li>(<u>Enter your client's requirement here</u>)</li>
</ol>
<br />
<div style="text-align: center;">
So when the disk-I/O throughput is PATHETIC, what does one do?...</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-JVjisO18cNQ/T9sIMYjSacI/AAAAAAAAA24/goAuZbMe5QQ/s1600/5-ways-optimise-hdd.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://4.bp.blogspot.com/-JVjisO18cNQ/T9sIMYjSacI/AAAAAAAAA24/goAuZbMe5QQ/s1600/5-ways-optimise-hdd.jpg" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">5 WAYS to optimise your HDD throughput!</td></tr>
</tbody></table>
<br />
<br />
<h2>
1. Bypass page-cache for "read-once" data.</h2>
<div style="text-align: justify;">
What exactly does page-cache do? It caches recently accessed pages from the HDD. Thus reducing seek-times for subsequent accesses to the same data. The key here being subsequent. The page-cache does NOT improve the performance the first time a page is accessed from the HDD. So if an app is going to read a file once and just once, then bypassing the page-cache is the better way to go. This is possible by using the <b>O_DIRECT</b> flag. This means that the kernel does NOT considered this particular data for the page-cache. Reducing cache-contention means that other pages (which wouold be accessed repeatedly) have a better chance of being retained in the page-cache. This improves the cache-hit ratio i.e better performance.</div>
<code class="tr_bq">
<br />
<span style="font-size: x-small;">void ioReadOnceFile()<br />
{<br />
/* Using direct_fd and direct_f bypasses kernel page-cache.</span>
<span style="font-size: x-small;"><br />
* - direct_fd is a low-level file descriptor<br />
* - direct_f is a filestream similar to one returned by fopen()<br />
* NOTE: Use getpagesize() for determining optimal sized buffers.<br />
*/<br /><br />
int direct_fd = open("filename", O_DIRECT | O_RDWR);</span>
<span style="font-size: x-small;"><br />
FILE *direct_f = fdopen(direct_fd, "w+");<br /><br />
/* direct disk-I/O done HERE*/</span>
<span style="font-size: x-small;"><br /><br />
fclose(f);</span>
<span style="font-size: x-small;"><br />
close(fd);<br />
}</span></code><br />
<br />
<h2>
2. Bypass page-cache for large files.</h2>
<div style="text-align: justify;">
Consider the case of a reading in a large file (ex: a database) made of a huge number of pages. Every subsequent page accessed get into the page-cache only to be dropped out later as more and more pages are read. This severely reduces the cache-hit ratio. In this case the page-cache does NOT provide any performance gains. Hence one would be better off bypassing the page-cache when accessing large files.</div>
<br />
<code class="tr_bq">
<span style="font-size: x-small;">void ioLargeFile()<br />
{<br />
/* Using direct_fd and direct_f bypasses kernel page-cache.</span>
<span style="font-size: x-small;"><br />
* - direct_fd is a low-level file descriptor<br />
* - direct_f is a filestream similar to one returned by fopen()<br />
* NOTE: Use getpagesize() for determining optimal sized buffers.<br />
*/<br /><br />
int direct_fd = open("largefile.bin", O_DIRECT | O_RDWR | O_LARGEFILE);</span>
<span style="font-size: x-small;"><br />
FILE *direct_f = fdopen(direct_fd, "w+");<br /><br />
/* direct disk-I/O done HERE*/</span>
<span style="font-size: x-small;"><br /><br />
fclose(f);</span>
<span style="font-size: x-small;"><br />
close(fd);<br />
}</span>
</code>
<br />
<br />
<h2>
3. If (cpu-bound) then scheduler == no-op;</h2>
<div style="text-align: justify;">
The io-scheduler optimises the order of I/O operations to be queued on to the HDD. As seek-time is the heaviest penalty on a HDD, most I/O schedulers attempt to minimise the seek-time. This is implemented as a variant of the elevator algorithm i.e. re-ordering the randomly ordered requests from numerous processes to the order in which the data is present on the HDD. require a significant amount of CPU-time.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Certain tasks that involve complex operations tend to be limited by how fast the cpu can process vast amounts of data. A complex I/O-scheduler running in the background can be consuming precious CPU cycles, thereby reducing the system performance. In this case, switching to a simpler algorithm like no-op reduces the CPU load and can improve system performance.</div>
<code class="tr_bq"><span style="font-size: x-small;">echo noop > /sys/block/<block-dev><block -device="-device">/queue/scheduler</block></span><br /><block -device="-device"></block></code><block -device="-device"><br />
</block><br />
<h2>
4. Block-size: Bigger is Better</h2>
<blockquote class="tr_bq">
<b>Q. How will you <a href="http://www.amazon.com/gp/product/B000JBY0RY/ref=as_li_qf_sp_asin_tl?ie=UTF8&tag=thec045-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=B000JBY0RY">move Mount Fuji</a><img alt="" border="0" src="http://www.assoc-amazon.com/e/ir?t=thec045-20&l=as2&o=1&a=B000JBY0RY" height="1" style="border: none !important; margin: 0px !important;" width="1" />
to bangalore?</b><br />
Ans. Bit by bit.</blockquote>
<div style="text-align: justify;">
While this will eventually get the job done, its definitely NOT the most optimal way. From the kernel's perspective, the most optimal size for I/O requests is the the filesystem blocksize (i.e the page-size). As all I/O in the filesystem (and the kernel page-cache) is in terms of pages, it makes sense for the app to do transfers in multiples of pages-size too. Also with multi-segmented caches making their way into HDDs now, one would hugely benefit by doing I/O in multiples of block-size.</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-K3fmQj6PRmo/T9sKTHLBvaI/AAAAAAAAA34/fzfOrGbu4kY/s1600/block-size-vs-throughput.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://3.bp.blogspot.com/-K3fmQj6PRmo/T9sKTHLBvaI/AAAAAAAAA34/fzfOrGbu4kY/s1600/block-size-vs-throughput.jpg" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Barracuda 1TB HDD : Optimal I/O block size 2M (=4blocks)</td></tr>
</tbody></table>
The following command can be used to determine the optimal block-size <br />
<code class="tr_bq"><span style="font-size: x-small;">stat --printf="bs=%s optimal-bs=%S\n" --file-system /dev/<block-dev></span></code><span style="font-size: x-small;"> </span><br />
<span style="font-size: x-small;"> </span> <br />
<block -device="-device">
</block>
<br />
<h2>
5. SYNC vs. ASYNC (& read vs. write)</h2>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-vsDYg0I-UnA/T9sKY45vDqI/AAAAAAAAA4A/wHnX4rU8SKw/s1600/disk-io-detailed-async.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://3.bp.blogspot.com/-vsDYg0I-UnA/T9sKY45vDqI/AAAAAAAAA4A/wHnX4rU8SKw/s1600/disk-io-detailed-async.jpg" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">ASYNC I/O i.e. non-blocking mode is effectively faster with cache</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div style="text-align: justify;">
<div style="text-align: justify;">
<block -device="-device"><u><b>When an app initiates a SYNC I/O read</b></u>, the kernel queues a read operation for the data and returns only after the entire block of requested data is read back. During this period, the Kernel will mark the app's process as blocked for I/O. Other processes can utilise the CPU, resulting in a overall better performance for the system.</block><br />
<block -device="-device"></block><br />
<block -device="-device"><u><b>When an app initiates a SYNC I/O write</b></u>, the kernel queues a write operation for the data puts the app's process in a blocked I/O. Unfortunately what this means is that the current app's process is blocked and cannot do any other processing (or I/O for that matter) until this write operation completes.</block><br />
<block -device="-device"></block><br />
<block -device="-device"><u><b>When an app initiates an ASYNC I/O read</b></u>, the read() function usually returns after reading a subset of the large block of data. The app needs to repeatedly call read() with the size of data remaining to be read, until the entire required data is read-in. Each additional call to read introduces some overhead as it introduces a context-switch between the userspace and the kernel. Implementing a tight loop to repeatedly call read() wastes CPU cycles that other processes could have used. Hence one usually implements blocking using select() until the next read() returns non-zero bytes read-in. i.e the ASYNC is made to block just like the SYNC read does.</block><br />
<block -device="-device"></block><br />
<block -device="-device"><u><b>When an app initiates an ASYNC I/O write</b></u>, the kernel updates the corresponding pages in the page-cache and marks them dirty. Then the control quickly returns to the app which can continue to run. The data is <a href="http://www.westnet.com/%7Egsmith/content/linux-pdflush.htm" target="_blank">flushed to HDD</a> later at a more optimal time(low cpu-load) in a more optimal way(sequentially bunched writes).</block><br />
<block -device="-device"></block><br />
<block -device="-device">Hence, <b>SYNC-reads</b> and <b>ASYNC-writes</b> are generally a good way to go as they allow the kernel to optimise the order and timing of the underlying I/O requests.</block><br />
<block -device="-device"></block></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
<block -device="-device"></block></div>
<div style="text-align: justify;">
<br />
<div style="text-align: justify;">
<block -device="-device">There you go. I bet you now have quite a lot of things to say in your presentation about improving disk-IO. ;-)</block></div>
<br />
<br />
<br />
<block -device="-device"><span id="goog_989967194"></span><span id="goog_989967195"></span>PS: If your client fails to comprehend all this (just like when he saw inception for the first time), then do not despair. Ask him to go buy a <a href="http://ark.intel.com/compare/66247" target="_blank">freaking-fast SSD</a> and he will never bother you again.</block><br />
<br />
<block -device="-device">More on <b><a href="http://thecodeartist.blogspot.com/2012/08/hdd-filesystems-osync.html" target="_blank">How data-barriers affect HDD Throughput and Data-integrity</a></b>. </block></div>
<block -device="-device"></block></div>
</div>
Unknownnoreply@blogger.com5tag:blogger.com,1999:blog-5879050571098780501.post-42161548031000927552012-04-30T01:05:00.000+05:302012-09-12T01:37:15.805+05:30Omnivison ov3640 i2c sccb<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-af92_nBJ4eQ/T52g0Et_U0I/AAAAAAAAA0U/AS2jsQN_j2U/s1600/ov3640-camera-module.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="233" src="http://1.bp.blogspot.com/-af92_nBJ4eQ/T52g0Et_U0I/AAAAAAAAA0U/AS2jsQN_j2U/s400/ov3640-camera-module.jpg" width="400" /></a></div>
<div style="text-align: justify;">
<h2 style="font-family: Verdana,sans-serif; text-align: center;">
TASK : Write a device-driver for Omnivision ov3640 camera</h2>
<h2 style="font-family: Verdana,sans-serif; text-align: center;">
Timeline : A.S.A.P (is there any other way? :P)</h2>
For the aptitude champs out there, here is a quick one:</div>
<div style="text-align: justify;">
<blockquote class="tr_bq">
Q. If you are sitting facing west and running I2C at 0.00000000055kbps and a bear appears in front of you, what color is the bear?</blockquote>
</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
Not sure? Read on...<br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span></div>
<div style="text-align: justify;">
</div>
<div style="text-align: left;">
<h2 style="font-weight: normal;">
<b><span style="font-family: "Courier New",Courier,monospace;">DAY 1 : Initial study</span></b></h2>
<span style="font-family: "Courier New",Courier,monospace;"></span></div>
<div style="text-align: justify;">
<blockquote class="tr_bq">
<b>A bit about ov3640</b>: The ov3640 (color) image sensor is a 1/4-inch 3.2-megapixel CMOS image sensor that is capable of QXGA(2048x1536)@15FPS using OmniPixel3™ technology in a small footprint package. It provides full-frame, sub-sampled, windowed or arbitrarily scaled 8-bit/10-bit images in various formats via the control of the <a href="http://www.google.co.in/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CDgQFjAB&url=http%3A%2F%2Fwww.ovt.com%2Fdownload_document.php%3Ftype%3Ddocument%26DID%3D63&ei=aZ6dT-KmO83orQfzg5xS&usg=AFQjCNFJ0PH4AGGT5wKG57iybhAq5qRj3w" target="_blank">Serial Camera Control Bus (SCCB)</a> interface or MIPI interface. It supports both a digital video parallel port and a serial MIPI port.</blockquote>
</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
Searching the "<a href="http://en.wikipedia.org/wiki/Internets" target="_blank">internets</a>", an old "v4l2-int" styled driver for ov3640 is available for the linux-kernel. This will have to do for now. Can scavenge the camera configuration register-settings from it.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The <a href="http://www.ovt.com/download_document.php?type=sensor&sensorid=7" target="_blank">Omnivision ov3640 product-brief</a> contains the following functional block-diagram of ov3640:</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-QfW7HFW-ly0/T52N7DSnsTI/AAAAAAAAAzc/ZFAPzPvqddk/s1600/ov3640-functional-block-diagram.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="386" src="http://1.bp.blogspot.com/-QfW7HFW-ly0/T52N7DSnsTI/AAAAAAAAAzc/ZFAPzPvqddk/s540/ov3640-functional-block-diagram.jpg" width="540" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The camera is controlled using the <a href="http://www.google.co.in/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CDgQFjAB&url=http%3A%2F%2Fwww.ovt.com%2Fdownload_document.php%3Ftype%3Ddocument%26DID%3D63&ei=aZ6dT-KmO83orQfzg5xS&usg=AFQjCNFJ0PH4AGGT5wKG57iybhAq5qRj3w" target="_blank">SCCB</a> bus. Again back to the "<a href="http://en.wikipedia.org/wiki/Internets" target="_blank">internets</a>". SCCB is an i2c-clone i.e a two-wire serial protocol that has significant differences from I2C to merits its own specification.</div>
<ol style="text-align: left;">
<li>According to spec, SCCB supports only upto 100Khz (not more).</li>
<li>I2C spec requires pullups with open-collector(drain) drivers everywhere. SCCB requires CMOS-like drivers which are always either +VDD or GND i.e no pullups.</li>
<li>In I2C, after every 8bits transferred, the 9th bit is designated ACK. The slave pulls SDA low to ack. SCCB designates the 9th bit "dont-care". SCCB spec states that the master continues regardless of ACK/NACK in the 9th bit.</li>
</ol>
<div style="text-align: justify;">
<br /></div>
<div style="font-family: "Courier New",Courier,monospace; text-align: left;">
<h2>
DAY 2 : First attempt</h2>
</div>
<div style="text-align: justify;">
Following the omnivision product-brief and the datasheet, the ov3640 camera-module is connected with the CPU as follows:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-7m3VxzTcdTA/T52Otr2wA5I/AAAAAAAAAzk/YPHry-jIxHg/s1600/ov3640-omap-interface-not-working.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-7m3VxzTcdTA/T52Otr2wA5I/AAAAAAAAAzk/YPHry-jIxHg/s1600/ov3640-omap-interface-not-working.png" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
So far SCCB looked to be a simpler less restrictive version of I2C. Having worked extensively on I2C previously, was under the impression that setting up the basic comunication between the CPU and ov3640 would be a walk in the park. Wrote a simple skeleton i2c-driver and registered it with the kernel. Scavenged the I2C read/write routines from the old ov3640 driver and booted-up the device...</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
...and the driver failed to load as I2C-read failed. The ID register of ov3640 did NOT match the expected ID. Inserting logs in the code showed that the I2C-read routine was failing. The CPU was NOT getting an ACK from the ov3640 sensor. A true WTF moment as the I2C routines in the driver were tested to be working properly in earlier devices.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Oh well, maybe should really not expect an ACK i suppose. What with ov3640 being an SCCB device and not I2C. Started digging into the I2C core driver. found a provision to ignore this NACK(absense of ACK from slave). Updated the ov3640 driver to set the IGNORE_NACK flag and tried again. Now the I2C-read routine completed successfully despite there being no ACK from the slave. But still the driver failed to load. Turns out the contents of the ID register, read over I2C, did NOT match the expected value. The I2C-read routine was returning the contents of the ID register as "0". Further debugging showed that attempting to read any register of ov3640 over I2C gave the same result- a nice big ZERO. It was evident now that something was terribly wrong.</div>
<div style="font-family: "Courier New",Courier,monospace; text-align: justify;">
<br /></div>
<div style="font-family: "Courier New",Courier,monospace; text-align: justify;">
<h2>
DAY 3 : Challenge Accepted</h2>
</div>
<div style="text-align: justify;">
Time to bring out the big guns. Switched to multimeter and oscilloscope. Tested the lines from CPU to the ov3640 connecters for proper continuity. Booted-up the device and probed the I2C lines. The master was sending in the right values alright. But ov3640 was simply not responding. Suspect no.1 ? the I2C slave-id.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Ov3640 spec mentions that it responds to 2 I2C slave-IDs. 0x78 & 0x79. Had tried 0x78 so far. 0x79 also makes no difference - still no data from ov3640. Further digging through the docs i find one interesting line which mentions that the addresses are special in the sense that 0x78 is used to write and 0x79 to read the device. Hmmm... interesting. Lokks like these are 8bit addresses including the read/write bit of I2C. Which means the actual device slave-id is just the 7MSBs (common to 0x78 & 0x79) i.e. 0x3C. Face-palm!</div>
<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-i4Od6f4D5Yw/T52UHOIZftI/AAAAAAAAA0I/w_vOptBot-w/s1600/i2c-address.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="91" src="http://2.bp.blogspot.com/-i4Od6f4D5Yw/T52UHOIZftI/AAAAAAAAA0I/w_vOptBot-w/s400/i2c-address.jpg" width="400" /></a></div>
<br /></div>
<div style="text-align: justify;">
Changed the slave-id of the ov3640-driver and booted-up the device, but still no dice. It would be easier to light a fire with 2 stones and twig.</div>
<div style="font-family: Georgia,"Times New Roman",serif; text-align: justify;">
<br /></div>
<div style="font-family: "Courier New",Courier,monospace; text-align: justify;">
<h2>
DAY 4 : ...and let there be light</h2>
</div>
<div style="text-align: justify;">
Lost all hopes of getting this to work. Swapped other camera modules. Tried a couple of other boards. But the ov3640 just does not seem to respond to anything. It is as if the module is not even powered-on.<br />
<br />
Maybe, mayyyybe thats wat it IS!</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Back to the schematics.</div>
<div style="text-align: justify;">
I2C-CLK? check.</div>
<div style="text-align: justify;">
I2C-DATA? check.</div>
<div style="text-align: justify;">
CAM_XCLK? check.</div>
<div style="text-align: justify;">
CAM-IO? check.</div>
<div style="text-align: justify;">
CAM-digital? check.</div>
<div style="text-align: justify;">
CAM-Analog? Do we really need to power the sensor array at this stage?</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Well what the heck nothing else seems to work anyway. Might as well try this. So quickly pulled down a line from an existing 3.3V power-rail on the board. Placed a diode along it to drop it down a bit and powered-on the board.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-eEs5Mqg08ds/T52PFXzfefI/AAAAAAAAAzs/llLqkY4QCoA/s1600/ov3640-omap-interface-working.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-eEs5Mqg08ds/T52PFXzfefI/AAAAAAAAAzs/llLqkY4QCoA/s1600/ov3640-omap-interface-working.png" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
And VOILA! It worked. The driver was able to read the ov3640 module properly.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-F81ol3dIT8Q/T52PeTxR6yI/AAAAAAAAAz0/auAohJ_P9DI/s600/ov3640-I2C-working-with-ACK.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="284" src="http://3.bp.blogspot.com/-F81ol3dIT8Q/T52PeTxR6yI/AAAAAAAAAz0/auAohJ_P9DI/s540/ov3640-I2C-working-with-ACK.png" width="540" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The ov3640 even responded to the default settings (QXGA@15FPS). Pretty neat eh?</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Oh well... sure makes me look foolish, now that it works. :-)<br />
Ah well there's always the first time for everything. ;-) ;-)</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
And now that it works, was able to summarise the following</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<h3>
Hardware connections:</h3>
</div>
<ul style="text-align: justify;">
<li><b>CAM-Analog</b> (2.8V for powering-on the module)</li>
<li><b>CAM-IO 1.8V</b> (1.8V for i2c-communication)</li>
<li><b>CAM-Digital</b> (1.5V generated by module)</li>
<li><b>I2C_CLK</b> (1.8V 400Khz-MAX for i2c-communication)</li>
<li><b>I2C_DATA</b> (1.8V bi-directional)</li>
<li><b>CAM_XCLK</b> (24Mhz reqd. for internal PLL)</li>
<li><b>CAM_PCLK</b> (generated by module)</li>
</ul>
<div style="text-align: justify;">
<h3>
Software configurations:</h3>
</div>
<ul style="text-align: justify;">
<li><b>I2C slave-id</b> 0x3c</li>
<li>ov3640 DOES provide an <b>ACK</b> (its I2C, NOT sccb)</li>
<li>Works on <b>I2C@400KHz</b></li>
</ul>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
With the above specs, surely this begs the question that why does someone go all the way to define their own bus-specs when the hardware obviously works on I2C!!!! WHY Omnivision? WHY????</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
<blockquote class="tr_bq">
Designing you own serial-bus? == $1,000,000<br />
<br />
Not using it in your own products? == $0<br />
<br />
The smile on my face when i finally figure it out? PRICELESS</blockquote>
</div>
<div style="text-align: justify;">
Somethings, money can't buy. For everything else , there is... Ah wait, i'm forgetting something now, right? Well here goes...</div>
<div style="text-align: justify;">
<br /></div>
<blockquote class="tr_bq">
<div style="text-align: justify;">
Q. If you are sitting facing west and running I2C at 0.00000000055kbps and a bear appears in front of you, what color is the bear?</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Ans. If it takes 4 days to transfer a byte, do you REALLY think i care!!</div>
</blockquote>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
NOTE:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
[i]. No bears were harmed in the development of this camera-module.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
[ii]. The image captured above did NOT appear out of the blue with only the driver in place. Several days of tweaking the exposure/white-balance settings and an earthquake later, managed to get the Kernel-driver, Android camera-HAL and app to work together.</div>
<div style="text-align: justify;">
<br /></div>
</div>
Unknownnoreply@blogger.com81tag:blogger.com,1999:blog-5879050571098780501.post-73114874993786151922012-04-06T20:00:00.000+05:302012-09-16T13:05:36.036+05:30Android Double-buffering, Page-Flip and HDMI<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="font-family: Verdana,sans-serif; text-align: center;">
<span style="font-size: small;">(a.k.a The case of the disappearing charging-icon)</span></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<span style="font-size: small;"><br /></span></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<span style="font-size: small;">The following is an account of the final development stages of an android phone, which for obvious reasons will not be named. The bug in itself was a very simple matter and the consequent fix too. But the entire process of discovering what exactly was happening was quite fun. (Ya right! tell that to my manager :P)</span></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<br /></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<a href="http://3.bp.blogspot.com/-Nn_od7TN5rI/TuYFLX9CoqI/AAAAAAAAAv4/pvpGEWsXzaU/s1600/sample-lockscreen.jpg" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-Nn_od7TN5rI/TuYFLX9CoqI/AAAAAAAAAv4/pvpGEWsXzaU/s1600/sample-lockscreen.jpg" /></a><span style="font-size: small;">The android phone, i was working on, supported connecting an external-display/TV via MHL(<a href="http://en.wikipedia.org/wiki/Mobile_High-definition_Link" target="_blank">Mobile-HD link</a> i.e. HDMI-over-USB). Once connected, the entire UI would be displayed on the TV. The phone battery would also charge while connected.</span></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<span style="font-size: small;"><br /></span></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<span style="font-size: small;">The issue in question was initially raised as an application issue. It so happened that the <u>charging status was not being displayed properly on the lockscreen</u>.</span><span style="font-size: small;"> </span></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<br /></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<h2>
<span style="font-size: small;">I. The issue...</span></h2>
<span style="font-size: small;">With the device locked, when a external MHL-cable was connected to the android phone, it used to update the charging-icon. But if removed immediately, the charging-icon would continue to be displayed. This would not happen always. But sometimes even upto a minute after the cable was removed, the status would continue to be displayed as "charging".</span></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<span style="font-size: small;"><br /></span></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<span style="font-size: small;">The application developers banged their heads over the weekend and finally pushed the issue onto the underlying kernel drivers, stating that they were updating the charging-status as-&-when they get an update from the framework, which in turn depends on the fuelguage/battery-driver to obtain the battery-status.</span></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<a href="http://2.bp.blogspot.com/-tAyvuujwPVQ/TuYFVoDEw9I/AAAAAAAAAwA/IweIuJ__mDo/s1600/bang+head+here.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="320" src="http://2.bp.blogspot.com/-tAyvuujwPVQ/TuYFVoDEw9I/AAAAAAAAAwA/IweIuJ__mDo/s320/bang+head+here.jpg" width="264" /></a><span style="font-size: small;"> </span></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<span style="font-size: small;">It was now the kernel developers turn to use the "stress-reduction kit". After hours of logging almost every single instruction in every single interrupt routine, it was quite evident that the battery driver was not at fault. It was promptly reporting the connect/disconnect events when(and only when) an MHL cable was inserted/removed. The android framework was getting the events and eventually the lockscreen application too.</span></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<span style="font-size: small;"><br /></span></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<span style="font-size: small;">So now the question was that if EVERYTHING was working as it was supposed to, why the charging-status was not being displayed correctly?</span></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<span style="font-size: small;"><br /></span></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<br />
<h2>
<span style="font-size: small;">II. The peculiar observation...</span></h2>
<span style="font-size: small;">The peculiar thing about the disappearing charging-icon was that it was almost never for the same amount of time. Every time we tested it by plugging-in the cable, if it would disappear, it would do so for varying periods of time and then appear again onscreen.</span><br />
<span style="font-size: small;"><br /></span><br />
<br />
<h2>
<span style="font-size: small;">III. What it meant...</span></h2>
<span style="font-size: small;">We finally got onto the right track after we saw that the icon ALWAYS re-appeared onscreen just as the clock on the lock-screen updated itself. As it turned out the culprit was the display driver. When plugging in the MHL cable, there was some amount of tinkering going on in the background to handle the multiple displays and/or switch to the secondary(external HDTV over MHL) from the primary(mobile-LCD). As is the norm, the display was double-buffered to improve performance and prevent onscreen flickering and tearing. Plugging-in the MHL-cable just as the display driver was initiating a <i>swapbuffer()</i> (i.e. a <a href="http://en.wikipedia.org/wiki/Multiple_buffering#Page_flipping" target="_blank">page-flip operation</a> to pick the back-buffer to display onscreen) the device would then initiate another <i>swapbuffer()</i> which meant the stale buffer was displayed onscreen. to add to the misery the </span><span style="font-size: small;">"smart" </span><span style="font-size: small;">display driver was programmed to skip redundant <i>swapbuffer()</i> calls. i.e. unless the display contents had changed from the time the previous call to <i>swapbuffer()</i> it would not refresh the display unnecessarily. This meant that after plugging-in the MHL-cable, once the wrong screen (one without the chargin-icon) was displayed, it would not be refreshed unless something else changed onscreen.</span><br />
<br />
<span style="font-size: small;">Usually the onscreen clock forced a refresh of the buffers when the time was updated. As it showed time only down to the minute, it would mean that sometimes the display could be "stale" for as long as (but no longer than) a minute. An</span> additional forced-refresh in the MHL-cable detection routine fixed the issue properly.<br />
<br />
A simple example of double-buffering is shown below:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-B4Tc-NMCbJg/TvWchYEOO2I/AAAAAAAAAwU/FosNLhgqnBw/s450/double-buffering.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-B4Tc-NMCbJg/TvWchYEOO2I/AAAAAAAAAwU/FosNLhgqnBw/s520/double-buffering.jpg" /></a></div>
<span style="font-size: small;"> </span><br />
<br />
<br />
<h2>
<span style="font-size: small;">IV. Could Triple-buffering have prevented this issue?</span></h2>
<blockquote class="tr_bq">
<span style="font-size: small; font-weight: normal;">Triple-buffering involves 2 back-buffers. at any given moment, the display-driver can immediately pick one</span><span style="font-size: small; font-weight: normal;"> that is not being updated by the graphics h/w</span><span style="font-size: small; font-weight: normal;"> to display into the front-buffer.</span></blockquote>
Triple buffering itself has 2 variants:<br />
<b>(A)</b> <b>Triple-buffering with no-sync.</b> In this method the back-buffers are alternately updated by the graphics-h/w as fast as it can. At each Vsync, the display driver picks one of the buffers which is currently not being written to and swaps it with the front-buffer.<br />
<br />
<b>(B)</b> <b>Triple-buffering with Vsync.</b> In this method, the back-buffers are updated by the graphics h/w as fast as it can. But the update stops if both the back-buffers are updated but have not been displayed in the front-buffer yet. The display-driver as usual swaps one of the back-buffers witht he fornt-buffer at each Vsync. at this point the previous front-buffer which is now a back buffer is considered "stale" and the graphics h/w fills it up with the updated frame.<br />
<br />
Triple-bufffering used could potentially correct the issue as one of the back-buffers would hold the properly updated screen data and it even if it was not picked-up right away, it would be picked immediately in the following next <span style="font-size: small;"><i>swapbuffer()</i> call. Also in double-buffering, the graphics h/w doesn't have to wait for access to the backbuffer till the <i>swapbuffer()</i> completes the flip operation between the front and back buffers. This is not the case in triple-buffering, thus allowing the graphics h/w to run at full throttle thereby reducing the time that either of the backbuffers contains stale display data.</span><br />
<span style="font-size: small;"><br /></span><br />
<span style="font-size: small;">Further reading: <a href="http://en.wikipedia.org/wiki/Multiple_buffering" target="_blank">A detailed description of double/triple buffering</a>.</span></div>
</div>Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-5879050571098780501.post-47744073253990488912012-02-11T17:14:00.000+05:302012-02-11T17:19:36.662+05:30[patch] [resend] Preparing a modified patch<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<span style="font-size: small;">In case of any collaborated project (eg. linux-kernel), often after submitting a patch for review, we often receive several comments and need to make appropriate changes and generate a new "version2" of the patch containing the changes. If we are using <a href="http://en.wikipedia.org/wiki/Git_%28software%29" target="_blank">Git</a> for revision control, then the entire process becomes a snap.</span></div>
<div style="text-align: justify;">
<br /></div>
<h2>
How to prepare a modified patch for resend in <u>5 easy steps</u>:</h2>
<br />
STEP1.<br />
<code class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">git rebase -i <commit-id-just-before-our-changes></span></code>
<br />
<br />
STEP2.<br />
As discussed in the review, make the new changes to the source-files.<br />
<br />
STEP3.<br />
<code class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">git add <modified-filenames></span></code>
<br />
<br />
STEP4.<br />
<code class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">git commit --amend</span></code><br />
(shows editor with original commit-msg)<br />
Edit the commit-msg (or leave as-is) and quit.<br />
New commit is generated in the place of old commit.<br />
<br />
STEP5.<br />
<code class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">git format-patch HEAD~1</span></code>
<br />
DONE!! New patch version2 is ready for review now. :-)<br />
<br />
<div style="font-family: Verdana,sans-serif;">
If we do a diff between the PREV and NEW patch, we can see :</div>
<div style="font-family: Verdana,sans-serif;">
+ Changes made after review.</div>
<div style="font-family: Verdana,sans-serif;">
+ Time-Stamp change.</div>
<div style="font-family: Verdana,sans-serif;">
+ Hash change.</div>
</div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5879050571098780501.post-61791439518102326182012-01-14T20:10:00.000+05:302019-09-23T20:28:26.310+05:30Android Sensors and Location based services<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: center;">
Here is the talk i presented about <u><br />Sensors and location based services on Android</u> </div>
<div style="text-align: center;">
at <a href="http://blog.blrdroid.org/" target="_blank">B.A.(U).G / BLR-DROID.</a> </div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
Felt awesome talking to people, answering Qs about <a href="http://thecodeartist.blogspot.com/2011/11/ngps-location-fix-without-gps.html" target="_blank">nGPS.</a>
</div>
<br />
<div style="text-align: center;">
<iframe allowfullscreen="" frameborder="0" height="413" marginheight="0" marginwidth="0" scrolling="no" src="//www.slideshare.net/slideshow/embed_code/key/1ooSuGh594veqL" style="border-width: 1px; border: 1px solid #ccc; margin-bottom: 5px; max-width: 100%;" width="500"> </iframe> <br />
<br />
Download <b><a href="http://www.slideshare.net/cvs26/sensors-and-location-based-services/download" target="_blank">Sensors and location based services on Android</a></b></div>
</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-5879050571098780501.post-43332068872079123012011-12-28T19:44:00.005+05:302012-09-16T13:05:36.019+05:30Booting Android completely over ethernet<div dir="ltr" style="text-align: left;" trbidi="on"><div dir="ltr" style="text-align: left;" trbidi="on"><div dir="ltr" style="text-align: left;" trbidi="on"><div dir="ltr" style="text-align: left;" trbidi="on"><div style="font-family: inherit; text-align: justify;"><span style="font-size: small;">When developing embedded-systems, initial development stages often involve huge number of "<b>Modify-Build-Flash-Test</b>" cycles. </span><span style="font-size: small;">Test-Driven-Development methodology further promotes this style of development. </span><span style="font-size: small;">This leads to a break in the "flow" at the <b>Flash</b> stage. Flashing the device with a newly built set of binaries interrupts the otherwise smooth "<b>Modify-Build-Test</b>" flow. Also errors tend to creep-in in the form of an older binary being copied/flashed, often causing confusion during debugging and endless grief to the developer.</span></div><div style="font-family: inherit; text-align: justify;"><br />
</div><div style="font-family: inherit; text-align: justify;"><span style="font-size: small;">A simple way to avoid this is to have the binaries on the host-machine (a PC) and boot the embedded device directly using those binaries. In case of Android embedded system development, these binaries are the Linux-Kernel and the Android filesystem image.</span></div><div style="font-family: inherit; text-align: justify;"><br />
<blockquote class="tr_bq"><u><span style="font-size: small;">Pre-requisites:</span></u><br />
<ul><li>The embedded device</li>
<li>A linux PC </li>
<li>Ethernet connectivity between the two</li>
</ul></blockquote><br />
<span style="font-size: x-small;"><b>NOTE</b></span>: Below listed parts 1, 2 & 3 involve setting-up the "host" Linux PC. Part 4 describes configuring the device to boot directly using the binaries present on the "host". It is assumed that a functional bootloader (u-boot) is present on the device (internal-flash/mmc-card) and that ethernet-support(either direct or over usb) is enabled.</div><div style="font-family: inherit; text-align: justify;"><span style="font-size: small;"> </span></div><div dir="ltr" style="text-align: left;" trbidi="on"><div dir="ltr" style="text-align: left;" trbidi="on"><div dir="ltr" style="text-align: left;" trbidi="on"><div dir="ltr" style="text-align: left;" trbidi="on"><h1><br />
<span class="mw-headline" id="Part1: Linux kernel over tftp">Part1: Linux kernel over tftp</span></h1><h2><span class="mw-headline" id="Install_tftpd_and_related_packages">1. Install tftpd and related packages</span></h2><pre><code><b>host-PC$ </b>sudo apt-get install xinetd tftpd tftp
</code></pre><br />
<h2><span class="mw-headline" id="Create_.2Fetc.2Fxinetd.d.2Ftftp">2. Create /etc/xinetd.d/tftp </span></h2><pre><code><b>host-PC$ </b>cat <<EOF | sudo tee /etc/xinetd.d/tftp
service tftp
{
protocol = udp
port = 69
socket_type = dgram
wait = yes
user = nobody
server = /usr/sbin/in.tftpd
server_args = /srv/tftp
disable = no
}
EOF
</code>
</pre><h2><span class="mw-headline" id="Make tftp-server directory">3. Make tftp-server directory</span></h2><code><b>host-PC$ </b>mkdir <tftp-server-path> <br />
<br />
<b>host-PC$ </b>chmod -R 777 <tftp-server-path> <br />
<br />
<b>host-PC$ </b>chown -R nobody <tftp-server-path> </code> <br />
<h2><span class="mw-headline" id="Start_tftpd_through_xinetd">4. Start tftpd through xinetd</span></h2><code><b>host-PC$ </b> sudo /etc/init.d/xinetd restart </code> <br />
This concludes the tftp part of the setup process on the host.<br />
<h1><br />
<span class="mw-headline" id="Android fs over NFS">Part2: Android fs over NFS</span></h1><h2><span class="mw-headline" id="Install nfs packages">1. Install nfs packages </span></h2><code><b>host-PC$ </b>sudo apt-get install nfs-kernel-server nfs-common </code> </div><br />
<h2><span class="mw-headline" id="Modify /etc/exports">2. Add this line to /etc/exports</span> </h2><code><rootfs-path> *(rw,sync,no_subtree_check,no_root_squash) </code> </div><br />
<h2><span class="mw-headline" id="Restart service">3. Restart service</span> </h2><code><b>host-PC$ </b>sudo service nfs-kernel-server restart </code> <br />
<h2><span class="mw-headline" id="Update exports for the NFS server">4. Update exports for the NFS server</span> </h2><code><b>host-PC$ </b>exportfs -a </code> <br />
<h2><span class="mw-headline" id="Check NFS server">5. Check NFS server</span> </h2><code><b>host-PC$ </b>showmount -e </code> </div><br />
If everything went right, the <rootfs-path> will be listed in the output of showmount. <br />
<h1><br />
<span class="mw-headline" id="Where to put the files">Part3: Where to put the files</span></h1><h2><span class="mw-headline" id="Linux Kernel uImage">1. Linux Kernel uImage</span></h2>On the "host" PC,<br />
Copy the Linux-Kernel <b>uImage</b> into <tftp-server-path><br />
<ul style="text-align: left;"></ul><h2><span class="mw-headline" id="Android rootfs">2. Android rootfs</span></h2>On the "host" PC,<br />
Copy the contents of the Android rootfs into <rootfs-path><br />
<ul style="text-align: left;"></ul><h1><br />
<span class="mw-headline" id="Configuring the bootloader">Part4: Configuring the bootloader</span></h1></div></div><h2><span class="mw-headline" id="Update bootargs">1. Update bootargs</span></h2>Connect the embedded device to the host-PC over ethernet (either directly or via a switch/router) and power it on. As shown below, configure the bootloader to pick-up the kernel from the host-PC over tftp and to mount the filesystem from the host-PC over NFS. As both support configuring a static-ip for the embedded-device or obtaining one dynamically using dhcp, 4 combinations are possible (2 shown below). <br />
<br />
<u>nfs(static-ip)</u><u> and </u><u>tftp(dhcp)</u><br />
<code><b>U-Boot# </b>setenv bootargs 'console=ttyO0,115200n8 androidboot.console=ttyO0 mem=256M root=/dev/nfs ip=<client-device-ip> nfsroot=<nfs-server-ip>:<rootfs-path> rootdelay=2' <br />
<br />
<b>U-Boot# </b>setenv serverip 'host-pc-ip'<br />
<br />
<b> U-Boot# </b>bootm <Load address></code><br />
<br />
<u>nfs(dhcp)</u><u> and</u><u> tftp(static-ip)</u><br />
<code><b>U-Boot# </b>setenv bootargs 'console=ttyO0,115200n8 androidboot.console=ttyO0 mem=256M root=/dev/nfs ip=dhcp nfsroot=<nfs-server-ip>:<rootfs-path> rootdelay=2'<br />
<br />
<b>U-Boot# </b>setenv serverip 'host-pc-ip'<br />
<br />
<b>U-Boot# </b>setenv ipaddr 'client-device-ip'<br />
<br />
<b>U-Boot# </b>tftp <br />
<br />
<b>U-Boot# </b>bootm <Load address> </code><br />
<br />
<h2><span class="mw-headline" id="Boot">2. Boot ;-)</span></h2></div><b> </b></div><div style="text-align: center;"><span style="font-family: inherit; font-size: small;"><b>Linux-Kernel loaded over tftp</b></span></div><div class="separator" style="clear: both; text-align: center;"><img border="0" src="http://4.bp.blogspot.com/-LUeMVw97tMQ/Tvx735Nf2cI/AAAAAAAAAwk/xmA5fnfGeyA/s500/dhcp-boot.jpg" /></div><br />
<div style="text-align: center;"><span style="font-family: inherit; font-size: small;"><b>Filesystem mounted over NFS</b></span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://www.blogger.com/blogger.g?blogID=5879050571098780501" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-9nT6viM535U/TvyIDfw0axI/AAAAAAAAAxA/ksZrs0fL6qM/s500/nfs-mount.jpg" /></a></div></div><div class="separator" style="clear: both; text-align: left;"><br />
</div><div class="separator" style="clear: both; text-align: left;"><br />
</div></div>Unknownnoreply@blogger.com5HAL 3rd Stage, New Thipasandra, Bengaluru, Karnataka, India12.971062185172919 77.65342674902342412.967906685172919 77.64610574902342 12.974217685172919 77.660747749023429tag:blogger.com,1999:blog-5879050571098780501.post-48085248631421637832011-12-01T08:32:00.000+05:302013-05-16T15:46:43.498+05:30Why __read_mostly does NOT work as it should<div dir="ltr" style="text-align: left;" trbidi="on">
</div>
<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: justify;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
In modern SMP(multicore) systems, any processor can write to a memory location. The other
processors have to update their caches immediately. For that reason, SMP systems implement the concept of "cacheline bouncing" to move "ownership" of cached-data between cores. This is effective but expensive.<br />
<br />
Individual cores have private L1 caches which are extremely faster than the L2 and L3 caches which are shared between multiple cores. Typically, when a memory location is going to be ONLY read repeatedly, but never written to (for example a variable tagged with the <i>const</i> modifier), each core on the SMP system can safely store its own copy of that variable in its private(non-shared) cache. As the variable is NEVER written, the cache-entry never gets invalidated or "dirty". Hence the cores never need to get into "cache line bouncing" for that variable.<br />
<br />
Take the case of the x86 architecture,<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><img border="0" height="320" src="https://lh6.googleusercontent.com/-PIETb3X6Qjk/Tt_Vdd4tfuI/AAAAAAAAAvI/3hf9nfocZuU/s480/intel-die-caches.jpg" style="margin-left: auto; margin-right: auto;" width="520" /></td></tr>
<tr><td class="tr-caption" style="text-align: center;">An Intel core i5 die showing the various caches present</td></tr>
</tbody></table>
<ul>
<li><span style="font-size: x-small;">[NON-SMP]</span> Intel Pentium 4 processor has to communicate between threads over
the front-side bus, thus requiring at least a 400-500 cycle delay.</li>
<li><span style="font-size: x-small;">[SMP]</span> Intel Core processor family allowed for communication over a shared
L2 cache with a delay of only 20 cycles between pairs of cores and the
front-side bus between multiple pairs on a quad-core design.</li>
<li><span style="font-size: x-small;"> [SMP]</span> The use of a
shared L3 cache in the Intel Core i7 processor means that going across a
bus to synchronize with another core is NEVER required unless a
multiple-socket system is being used.</li>
</ul>
The copies of "read-only" locations usually end-up being cached in the private caches of the individual cores, which are several orders of magnitude faster than the shared L3 cache.</div>
<br />
<br />
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<span style="font-size: large;">How <i>__read_mostly</i> is <b>supposed</b> to work:</span> </div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
When a variable is tagged with the<i> __read_mostly</i> annotation, it is a signal to the compiler that accesses to the variable will be mostly reads and rarely(but NOT never) a write.<br />
<br />
All variables tagged <i>__read_mostly</i> are grouped together into a single section in the final executable.
This is to improve performance by allowing the system to optimise access time
to those variables in SMP systems by allowing each core to maintain its own copy of them variable in it local cache. Once in a while when the variable does get written to, "cacheline bouncing" takes place. But this is acceptable as the the time spent by the cores constantly synchronising using locks and using the slower shared-cache would be far more than the time it takes for the multiple cores to operate on own copies in their independent caches.<br />
<br />
<span style="font-size: large;">What <i><b>actually</b></i> happens: </span><br />
<div style="text-align: left;">
<span style="font-size: x-small;"><span style="font-size: x-small;"><span style="font-size: x-small;">(</span>NOTE: I</span>n the following section, </span><span style="font-size: x-small;"><span style="font-size: x-small;">"<span style="font-size: x-small;">e</span>lements"</span> refers to memory<span style="font-size: x-small;"> block</span>s<span style="font-size: x-small;"><span style="font-size: x-small;"> </span>which <span style="font-size: x-small;">are </span></span></span><span style="font-size: x-small;"><span style="font-size: x-small;"><span style="font-size: x-small;"><span style="font-size: x-small;"><span style="font-size: x-small;"><span style="font-size: x-small;">are smaller<span style="font-size: x-small;"> </span>than <span style="font-size: x-small;">a<span style="font-size: x-small;"> single</span></span> cache-line and<span style="font-size: x-small;"> </span>are so sparse <span style="font-size: x-small;">in main-me<span style="font-size: x-small;">mo</span>ry <span style="font-size: x-small;">that<span style="font-size: x-small;"> </span>a single</span></span></span></span></span></span> cache-line<span style="font-size: x-small;"> <span style="font-size: x-small;">can<span style="font-size: x-small;">not</span></span> contain</span></span> them both simultaneously.<span style="font-size: x-small;">)</span></span></div>
The problem with the above approach is that once all the <i>__read_mostly</i> variables are grouped into one section, the remaining "non-read-mostly" variables end-up together too. This increases the chances that two frequently used elements (in the "non-read-mostly" region) will end-up competing for the same position (or cache-line, the basic fixed-sized block for memory<-->cache transfers) in the cache. Thus frequent accesses will cause excessive cache thrashing on that particular cache-line thereby degrading the overall system performance.<br />
<br />
This situation is slightly alleviated by the fact that modern cpu caches are mostly 8way or 16way set-associative. In a 16way associative cache, each element has a choice of 16 different cache-slots. This means that two very frequently accessed elements, though closely located in memory, can still end-up in 2 different slots in the cache, thereby preventing cache-thrashing (which would have occurred had both continued competing for the same cache-line slot). In other words a minimum of 17 elements frequently accessed and closely located in memory are required for 2 of them to begin competing for a common cache-line slot.<br />
<br />
While this is true in the case of INTEL and its x86 architecture, ARM still sticks to 4way & 2way set-associative caches even in its Cortex A8, which means that just 3 or 5 closely located, frequently accessed elements can result in cache-thrashing on an ARM system. (<span style="font-size: x-small;"><b>Update</b></span>: <a href="http://thecodeartist.blogspot.com/2011/12/why-readmostly-does-not-work-as-it.html?showComment=1368693290291#c5777583867429395138" target="_blank">"Anonymous" rightly points out in the comments</a> that 16-way set associative caches have made their way into modern SoCs, ARM Cortex A9 onwards.)<br />
<br />
<span style="font-size: x-small;"><b>kernel/arch/x86/include/asm/cache.h</b> contains</span><br />
<code class="tr_bq">
<span style="font-size: x-small;">#define __read_mostly __attribute__((__section__(".data..read_mostly"))) </span></code>
<br />
<b><span style="font-size: x-small;">kernel/arch/arm/include/asm/cache.h:</span></b><span style="font-size: x-small;"> does NOT, thereby defaulting to the empty definition in</span><br />
<b><span style="font-size: x-small;"> kernel/include/linux/cache.h</span></b><br />
<code class="tr_bq">
<span style="font-size: x-small;">#ifndef __read_mostly<br />#define __read_mostly<br />#endif</span><b><span style="font-size: x-small;"> </span></b></code>
<b><span style="font-size: x-small;"> </span></b><br />
<br />
<span style="font-family: Verdana,sans-serif;"><u><b>UPDATE</b></u>: The patch </span>daf8741675562197d4fb4c4e9d773f53494203a5<span style="font-family: Verdana,sans-serif;"> enables </span><span style="font-family: Verdana,sans-serif;">support for </span><span style="font-family: Verdana,sans-serif;">__read_mostly in the linux kernel for ARM architecture as well.</span><br />
<br />
The reason for this? It turns out that most modern ARM SoCs have started
using 8/16-way set associative caches. For example, the <a href="http://infocenter.arm.com/help/topic/com.arm.doc.ddi0246a/DDI0246A_l2cc_pl310_r0p0_trm.pdf" target="_blank">ARM PL310 cache controller</a> (as <a href="http://thecodeartist.blogspot.com/2011/12/why-readmostly-does-not-work-as-it.html?showComment=1368693290291#c5777583867429395138" target="_blank">"Anonymous" rightly points out in the comments</a>)
available on the ARM Cortex-A9 supports 16-way set associativity. The
above patch now makes sense on modern ARM SoCs as the probability of
cache-thrashing is reduced by the larger "N" in the N-way associative
caches.<br />
<br />
With the number of cores increasing rapidly and the on-die cache size growing slowly, one must always aim to:<br />
<ul>
<li>Minimise access to
the last level of shared cache to improve performance on multicore systems.</li>
<li>Increase associativity of private caches (of individual cores) to eliminate cache-slot contention and reduce cache-thrashing.</li>
</ul>
<ul>
</ul>
</div>
</div>
Unknownnoreply@blogger.com5tag:blogger.com,1999:blog-5879050571098780501.post-48635161133113309162011-11-28T14:46:00.001+05:302012-01-17T18:04:21.911+05:30nGPS : Location fix without GPS<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="font-family: Verdana,sans-serif; text-align: center;">
<span style="font-size: x-small;"><i>NOTE: Skip this post if you do NOT live on planet earth.</i></span></div>
<div style="font-family: Verdana,sans-serif;">
<span style="font-size: x-small;"><i><br /></i></span></div>
<div style="font-family: Verdana,sans-serif; text-align: justify;">
<span style="font-size: x-small;"><i>This is one of the ideas that i hit upon when preparing for a talk <a href="http://thecodeartist.blogspot.com/2011/11/sensors-on-android-droidcon2011.html" target="_blank">Sensors on Android @DroidCon2011</a>. It is an unusual application of the on-board sensors present most Android devices. Due to lack of time i was unable to present it in much detail during my talk. So here goes...</i></span></div>
<div style="font-family: Verdana,sans-serif;">
<span style="font-size: small;"><br /></span></div>
<div style="font-family: Verdana,sans-serif;">
<span style="font-size: small;"><span style="font-size: large;"><b>nGPS</b></span> (NO GPS) is a way of obtaining a location fix without using any <a href="http://en.wikipedia.org/wiki/Global_Positioning_System" target="_blank">GPS</a>, <a href="http://en.wikipedia.org/wiki/Assisted_GPS" target="_blank">AGPS</a>, <a class="mw-redirect" href="http://en.wikipedia.org/wiki/Wi-Fi_Positioning_System" title="Wi-Fi Positioning System">Wi-Fi Positioning</a> and <a href="http://en.wikipedia.org/wiki/Mobile_phone_tracking" title="Mobile phone tracking">cell-site triangulation</a> technologies.</span></div>
<div style="font-family: Verdana,sans-serif;">
<span style="font-size: small;"><br /></span></div>
<blockquote class="tr_bq" style="font-family: Verdana,sans-serif;">
<div>
<u><span style="font-size: small;">Why would anyone want to use nGPS</span></u></div>
<div>
<span style="font-size: small;">- Pure GPS based systems take upto 10mins for 1st fix.</span></div>
<div>
<span style="font-size: small;">- AGPS, Wi-Fi positioning require an active data-connection.</span></div>
<div>
<span style="font-size: small;">- Cell-site triangulation requires network coverage.</span></div>
</blockquote>
<div style="font-family: Verdana,sans-serif;">
<span style="font-size: small;"><br /></span></div>
<div style="font-family: Verdana,sans-serif;">
<span style="font-size: small;">So without any of these technologies at our disposal, how do we obtain a "location-fix" i.e. a latitude-longitude pair representing our current position. The answer lies in the magnetic-field sensor.</span></div>
<div style="font-family: Verdana,sans-serif;">
<span style="font-size: small;"><br /></span></div>
<div align="justify" style="font-family: Verdana,sans-serif;">
<span style="font-size: small;">The Earth's magnetic field, as measured by a magnetic sensor on the Earth's surface, is combination of of several
magnetic fields generated by various sources. These fields interact with each other and the net resultant what the magnetic sensor measures. </span></div>
<div class="separator" style="clear: both; font-family: Verdana,sans-serif; text-align: center;">
<span style="font-size: small;"><a href="http://cgz.e2bn.net/e2bn/leas/c99/schools/cgz/accounts/staff/rchambers/GeoBytes%20GCSE%20Blog%20Resources/Images/Plate%20Tectonics/Plate%20Tectonics/ConvectionCurrent_labelled.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"></a></span></div>
<div class="separator" style="clear: both; font-family: Verdana,sans-serif; text-align: center;">
<span style="font-size: small;"></span></div>
<div class="separator" style="clear: both; font-family: Verdana,sans-serif; text-align: center;">
<span style="font-size: small;"></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://upload.wikimedia.org/wikipedia/commons/thumb/4/44/Structure_of_the_magnetosphere_mod.svg/669px-Structure_of_the_magnetosphere_mod.svg.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="153" src="http://upload.wikimedia.org/wikipedia/commons/thumb/4/44/Structure_of_the_magnetosphere_mod.svg/669px-Structure_of_the_magnetosphere_mod.svg.png" width="200" /></a></div>
<div style="font-family: Verdana,sans-serif; text-align: left;">
<span style="font-size: large;"><u>World Magnetic Model (WMM)</u></span></div>
<div style="font-family: Verdana,sans-serif; text-align: left;">
<span style="font-size: small;">Major contributors to a magnetic-field:</span><span style="font-size: small;"> </span></div>
<div style="font-family: Verdana,sans-serif; text-align: left;">
<span style="font-size: small;">+ Conducting, fluid outer core.</span><br />
<span style="font-size: small;">+ Earth's crust and upper mantle.</span><br />
<span style="font-size: small;">+ Electrical currents in the atmosphere.</span><span style="font-size: small;"> </span></div>
<div style="font-family: Verdana,sans-serif; text-align: left;">
<span style="font-size: small;">+ Local magnetic interference.</span></div>
<div style="font-family: Verdana,sans-serif; text-align: left;">
</div>
<div style="font-family: Verdana,sans-serif; text-align: left;">
<span style="font-size: small;">By filtering the local magnetic interference due to other electronic/electrical devices, we have a unique magnetic-field signature present at each place on earth. The WMM aims to provide an accurate estimate of this field. A device (having a magnetic sensor) can measure the components of this field. Then comparing it with the WMM values of the earth's field, one can identify the latitude/longitude of the present location.</span></div>
<div style="font-family: Verdana,sans-serif; text-align: left;">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://www.ngdc.noaa.gov/geomagmodels/IGRFWMM.jsp?defaultModel=WMM" style="margin-left: 1em; margin-right: 1em;" target="_blank"><img border="0" src="http://3.bp.blogspot.com/-POMJnPFfqbU/TtOAP0xMLEI/AAAAAAAAAuk/hJDKb9opCik/s1600/wmm.png" /></a></div>
<span style="font-size: small;"></span></div>
</div>
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;">Android contains built-in support for the WMM using the </span><b style="font-family: Verdana,sans-serif;"><a href="http://developer.android.com/reference/android/hardware/GeomagneticField.html" target="_blank">GeomagneticField</a> </b></span><span style="font-family: Verdana,sans-serif; font-size: small;">class. The GeomagneticField class utilises the WMM internally to provide an estimated magnetic field at any given point on
Earth at a given time. The important thing to note is that this class accepts the location (alongwith altitude and time) and provides the expected magnetic-field at that position (at that particular altitude and instant of time).</span><br />
<div style="font-family: Verdana,sans-serif;">
<br /></div>
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;">To determine the location using the GeomagneticField class, requires some reverse-lookup trickery on our part. More on it in another post.</span></span><br />
<br />
<span style="font-size: small;"><span style="font-family: Verdana,sans-serif;"><u>UPDATE</u> : A recent <a href="http://thecodeartist.blogspot.com/2012/01/android-sensors-and-location-based.html" target="_blank">talk on Sensors and Location based services on Android</a> at <a href="http://www.meetup.com/blrdroid/events/47569402/?a=ea1_lnm&rv=ea1" target="_blank">blr-droid meetp#12</a> featuring nGPS among other things.</span></span></div>Unknownnoreply@blogger.com18tag:blogger.com,1999:blog-5879050571098780501.post-59880453387431182652011-11-22T14:22:00.001+05:302011-11-22T14:36:35.458+05:30Tonight's the night : ICS<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="font-family: Verdana,sans-serif;">
<span style="font-size: small;">Two weeks after ICS was released, finally synced-up the entire source. 6GB.<br />Gave a repo sync and woke-up and it was done! Sweet na?...</span></div>
<div style="font-family: Verdana,sans-serif;">
<span style="font-size: small;"><br /></span></div>
<div style="font-family: Verdana,sans-serif;">
<span style="font-size: small;">A good thing with ICS is that pandaboard is supported as-is.<br />This means that i can</span><span style="font-size: small;">... </span></div>
<blockquote class="tr_bq">
<div style="font-family: "Courier New",Courier,monospace;">
<span style="font-size: small;">$ source build/envsetup.sh<br />$ lunch full_panda-eng<br />$ make -j4</span></div>
</blockquote>
<div style="font-family: Verdana,sans-serif; text-align: right;">
<span style="font-size: small;"> ...and i'm done! :-)</span></div>
<div style="text-align: right;">
<div style="text-align: left;">
<pre style="font-family: Verdana,sans-serif;"><span style="font-size: small;">And so i have. Estimating 4hrs for a build on my "old" pc.
Tonight's the night... ;-)</span></pre>
</div>
</div>
</div>Unknownnoreply@blogger.com0