Only two days ago the contact messaging application Twitter suffered another bout of downtime, leaving some users frustrated and others asking why the platform continues to suffer problems.
Techcrunch recently spoke to an individual who is familiar with the technical problems at Twitter as well as the challenges that lay ahead for the startup. He re-iterated his belief that the problems lay not with Blaine Cook (the former head of engineering who was shown the door), nor with NTT (their host) but with the early lack of understanding of how complex their problems would be.
The issue is that group messaging is very difficult to achieve at a grand scale. Other large sites such as WordPress and Digg are mostly dealing with known problems, such as how to serve a large number of pages or a large number of images. Twitter is unique in that it needs to parse a large number of messages and deliver them to multiple recipients, with each user having unique connections to other users.
Social networks have similar complexity issues, but they only usually need to route a message to a single user (or at the most to a defined group). Even so, social networks like Friendster struggled for years with technical and scaling issues. Twitter is specifically dealing with text messages, and in most cases with active users those messages are very frequent and go out to hundreds of contacts (or followers, as they are referred to in Twitter). Every new Twitter user and every new connection results in an exponentially greater computational requirement.
Some of the best web applications are able to efficiently solve very complex problems to produce simple results for users (Eg. Google). The success of these applications is due to the innovative efforts by developers to solve large technical challenges, where they have often had to break new ground for solutions. For Twitter to reach a similar point of reliability they too will need a very comprehensive, ground-breaking solution.
The source that I spoke to also commented on how ill-prepared the Twitter team were and are for their current and future challenges. The small team contains a handful of engineers, with only a person or two committed to infrastructure and architecture. He goes on to point out that at Digg the team for network and systems alone is bigger than the total engineering team at Twitter, and that at Digg they are lead by well-known “A-list rockstars”.
The problems at Twitter are often attributed to their use of RubyOnRails, a web development framework. Twitter is almost certainly the largest site running on Rails, so fans of the framework and its developers have been quick to deflect the criticism and point it back at the engineers at Twitter. Utilizting a framework that has never conquered large-scale territory must certainly add to the risk and work required to find a solution. As an out-of-the box framework, Rails certainly doesn’t lend itself to large-scale application development.
Rails enabled Twitter to be developed quickly, to get to launch quickly and then to improve with new features relatively rapidly also. But the old adage of “Good, Fast, Cheap – pick two” certainly applies and Rails would do itself no harm by conceding that it isn’t a platform that can compete with Java or C when it comes to intensive tasks. Twitter is at a cross-roads as an application and Rails has served its purpose very well to date, but you are unlikely to see a computational cluster built with Ruby at Apache any time soon.
What we see at Twitter today is a very useful and popular service, but one with very complex underlying technical challenges to overcome. Twitter will require not only a new architecture approach and a big injection of the best minds they can find ($15 million can help), but will also need a little patience from users and those of us observing.
Read the rest of this entry »
Posted in Internet, Programming, Web 2.0 | 1 Comment »
YouTube has refused a request from U.S. Sen. Joe Lieberman (ID-Conn.) to remove all videos sponsored by terrorist organizations like Al-Qaeda, contending that most of them don’t violate its community guidelines.
Lieberman, chairman of the Senate Committee on Homeland Security and Governmental Affairs, Monday called on the Google Inc. subsidiary to remove video content produced by terrorist organizations that showed assassinations, deaths of U.S. soldiers and civilians, weapons training, “incendiary” speeches and other material intended to “encourage violence against the West.”
“Islamist terrorist organizations use YouTube to disseminate their propaganda, enlist followers, and provide weapons training,” Lieberman said in a letter to Google CEO Eric Schmidt. “YouTube also, unwittingly, permits Islamist terrorist groups to maintain an active, pervasive, and amplified voice, despite military setbacks or successful operations by the law enforcement and intelligence communities.”
In the letter, Lieberman noted that while YouTube posts community guidelines for its users, it does not appear that the company follows the guidelines. For example, he noted that despite rules that prohibit gratuitous violence on the site, there are videos of Al-Qaeda attacks on U.S. forces in which some soldiers are killed or injured.
When contacted, Google pointed to a YouTube blog post that said the company has removed some of the videos cited by Lieberman, primarily because they depicted gratuitous violence, advocated violence or used hate speech. However, the post also noted that most of the videos in question remain on the site “because they do not violate our community guidelines.”
“Hundreds of thousands of videos are uploaded to YouTube every day,” the YouTube blog post said. “Because it is not possible to pre-screen this much content, we have developed an innovative and reliable community policing system that involves our users in helping us enforce YouTube’s standards. Millions of users report potential violations of our community guidelines.”
YouTube went on to say that it encourages free speech and defends the right of its users to express unpopular points of view
“We believe that YouTube is a richer and more relevant platform for users precisely because it hosts a diverse range of views, and rather than stifle debate we allow our users to view all acceptable content and make up their own minds,” the company said. “Of course, users are always free to express their disagreement with a particular video on the site, by leaving comments or their own response video. That debate is healthy.”
Mark Hopkins, a blogger for Mashable, noted that YouTube has been “capricious and arbitrary” in deciding what content promotes hate speech or violence and should be removed. For example, he pointed out that YouTube took down a video showing victims of a Muslim terrorist attack, but allowed videos of homeless people who were paid to beat each other. A video of clothed females in Hong Kong with derogatory music towards women being played in the background was removed, while a video of a strip tease with nudity was allowed to remain on the site, he noted.
“[Lieberman's] primary concerns weren’t the usual suspects when you think of the things that American politicians find objectionable (rap music, graphic portrayals of violence, Grand Theft Auto and Janet Jackson’s nipple),” Hopkins noted.
Instead Lieberman brought up a topic that YouTube should be called on — allowing itself to be a participant in the dissemination of propaganda videos produced by Islamic terrorist organizations, Hopkins said. “If YouTube can spend millions enforcing DMCA and piracy concerns, they can take a few minutes and respond to valid citizen complaints against usage of the system to promote terrorist organizations,” he added.
Read the rest of this entry »
Posted in Google, Internet, Web 2.0 | No Comments »
MySpace thought it was all over when it secured the MySpace.co.uk domain in February this year. A decision by Nominet’s dispute resolution service handed over the address, which previously had been owned by a small UK ISP since 1997, two years before MySpace.com launched. But an appeals panel has today handed the domain back to Total Web Solutions (TWS), a company in Stockport, near Manchester.
The fact that Myspace.co.uk was originally used to offer email services and websites to subscribers meant TWS had insulated itself from an action for some time. But MySpace’s main argument to Nominet centred on the most recent use of the domain as a Pay Per Click website which sent MySpace.co.uk visitors to a parked page with advertisements for social networking websites including MySpace. MySpace Inc says the practice started in July 2005 when News Corp took it over, boosting its fame, but TWS claims it was “at least” before June 2005.
Secondly, at issue was whether parking the .co.uk domain had become “abusive” when the PPC ads changed because MySpace.com became well known. In the case of MySpace.co.uk, the ads on the parked domain did change to “reflect the fame of MySpace.com”, admitted TWS, “but that had happened automatically as a result of the algorithms used by parking company Sedo.” In other words, TWS fingered the firm servicing the ads. While MySpace Inc. argued that TWS should have exercised control over the content of the adverts, TWS said this did not constitute a “change of use”.
The three-person appeal panel said they were “reluctant to place any duty on a registrant, who has merely had the good fortune (or maybe ill fortune) to register a name in good faith…” so long as they don’t exploit the situation.
There appears to be no more steps that MySpace can take within the Nominet DRS arbitration process to challenge TWS’s right to hold onto the name. So it’s the end of the line – unless there is further action MySpace can take through the civil courts.
Total Web Solutions also claims that Nominet tried to “unfairly help” MySpace by at first denying the existence of emails sent between solicitors and MySpace which may have aided TWS’s case. The solicitor who represented Total Web Solution in the case, Jim Davies, is now standing for election to the Nominet board, as he believes it’s unwise to “operate the DRS (Domain Resolution Service) from within the company.” Davies has been involved in a number of the more high profile domain name disputes in the UK recently.
Total Web Solutions’ Managing director Paul Fallon issued a statement saying “We refused to be bullied by one of the largest media organisations in the world. This has been a very stressful case for a legitimate medium sized ISP to have to take on – but we had to defend our reputation and to stand up for what was right.”
Of course, the MySpace.co.uk domain is now effectively worthless since TWS would be ill-advised to do anything with it at all now. It is currently displaying a blank page. MySpace continues to use uk.myspace.com/. A MySpace spokesperson declined to comment.
Read the rest of this entry »
Posted in Domains, Internet, Legal, Web 2.0 | No Comments »
