Archive for the ‘ Internet and the Cloud ’ Category

How Google Taught Me to Hate The Cloud

Well, that was quick.  I guess.

As I tend to ramble on about here, I am a fickle indecisive mess when it comes to how to best manage all of my digital data.  I have been pushing more and more to "The cloud", cloud services, cloud hosting, cloud cloud cloud. 

I secretly hate the phrase "The Cloud" by the way.  I despise it.  It’s "Online" or "The Internet", the cloud is a really annoying buzzword.  I believe it stems somewhat from The internet being depicted as a could on line diagrams.  This comes from drawing network maps and wanting an easy way to represent the World Wide Web.  The Internet isn’t a cloud really at all.  It’s basically a huge clump of fibers and copper hooked between a shitload of routers of various sizes transmitting flashes of light and electricity between each other.  The Web is a better analogy then the Cloud but spiders are creepy.  Hell, calling it "A Series of Tubes" is actually more accurate than a cloud.  Even the RF involved with WiFi isn’t a disperes clump of like molecules floating in space.  Those transmissions are still a "virtual series of imaginary tubes" at their core.

But whatever.

The point is, I have been heavily advocating the use of the Internet for a while.  Maybe I’m just getting tired of it, but I find lately I care less and less about it.  This has permeated all aspects of what I do for my hobbies and non work life.  Despite the best efforts of loads of Internet Citizens, the web is slowly transforming from the crazy fun Wild Wild West into some sort of locked down internment camp.

There are ads everywhere.  i keep getting emails about "leaks" and "hacks" at websites that I’ve used in the past.  There is increasing tension around the idea of corporate and government surveillance of the bits in the name of stopping piracy or terrorism.   It’s just quickly becoming a scary mess.

Then there is the closing of Google Reader.

I’ve experienced sites being closed on me before.  I generally don’t care since it’s a service I never used (most likely because no one else was using it this, why it was closing).  I’ve even experienced this via Google.  I used to use Google Tasks and Google Wave, both have been gone for a while.  But Wave was gimmicky and a limited option and was eventually superseded by Google Docs.  Tasks wasn’t a major loss, there are probably a hundred different Tasks Manager methods and programs out there, many of which are superior to Google Tasks.

There are no good Google Reader alternatives.  Maybe there will be, but there isn’t anything quite as good.  I do not want to use Google Plus or replace my RSS with a bunch of cluttered Twitter and Facebook feeds (which often contain extraneous microblog posts and clutter my own stream). 

I also really like and need RSS.  I like small time blogs.  i don’t have time to regularly check to see if Bob’s Toy Blogger updated it’s once a month update, but I can add it to my RSS feed and wait for an update to slide in.  i feel like Google’s killing of RSS not only hurts the people who rely on it to get news, but it hurts the small time people like myself who rely on it to make it convenient to get their posts out. 

I can see where Google is going here.  It’s all Google+ now.  Reader will be replaced with a cluttered filtered Google Plus stream that is useless.  The real take away I’ve gotten is even Google, whom i relied on above all else, is not reliable to be there.

It makes me question my current online backup strategy completely.  I’ve already consolidated a lot fo my document level data into Evernote.  My Google Docs is empty, I feed news articles I want to keep into Evernote.  I store bits of code for the few times I ever do any coding.  I save stories and my own writing there.  It’s all there.  I realize this is kind of an "all in one basket" mindset that may not be good either.  There are several ways to backup Evernote however and, more importantly, Evernote is in the business of selling note taking software.  They don’t have social ads and email and virtual drives and video hosting and a thousand services, they pretty much just do <hankhill>Evernote and Evernote Accessories<hankhill>.

Also it means less data in the hands of Google, which is becoming increasingly less trustworthy.

There is some residual distrust created as well.  I’m using Skydrive to backup files, but Windows 8 is universally loathed and Office 365 is just as overpriced as normal Office.  I know Microsoft is a huge company that probably isn’t going anywhere anytime soon, but I do foresee them floundering a bit in the coming future as the wheels come off the mistake that was Windows 8.  I can see a service that is probably mostly a money loser like Skydrive hitting the cutting room floor when the shit hits the fan over there.

I’ve been pretty loyal to Flickr for years and I’ve found some tools to automate bulk uploads to use it as a secondary backup but Yahoo is a company that’s more of a mess than any other large tech company I can think of and Flickr is a definitely money loser for them (high bandwidth + dwindling user base is always a problem).

So one additional little bit fueling my Cloud distrust comes from my recent push to get more organized with my data.  I’ve been going through my thousands of book marks, sorting them into an online Delicious clone I set up or clipping them to Evernote for archival.  This of course leads to some dead links.  It’s a subtle reminder that this data will not be there forever.

A Tale of Two Cloud Services

box I’ve been going on and on lately about my "push to the cloud" and all that quite a bit lately.  I have come to a few roadblocks but it’s been going pretty well.  I’ve been adding Dropbox to the mix as well, as a supplement to Skydrive.  Why do I need Dropbox when I’m paying for Skydrive?  Actually there are quite a few good reasons to spread out a bit.

I think my primary reason, I fear being locked out of my Skydrive account over stupidity the way Youtube was trying to do.  I don’t plan to use either for any sort of piracy or illegal file distribution, but I have a fear of the "False Positive".  I don’t know that Microsoft scans files, but I suspect they do.  I only have one article I’m going on but there has been at least one case where a guy got his account suspended because he was storing pornography on his Skydrive.  Not distributing, just storing.  I don’t have any porn to put on Skydrive but I don’t really want the robot scanner to flip through say, my eBooks and say "oh hey, this file may be illegitimately gotten" and lock me out of all of my important files.

I have more confidence that this won’t happen on Dropbox.

I also really like Dropbox’s app compatibility.  For example, i have linked my Dropbox account to my O’Reilly account and now all of my purchases from O’Reilly automatically dump into my Dropbox.  I’ve set up a synced folder on Dropbox that I can dump PDFs into and Evernote will pull them in as new notes.  I’ve got a watch folder going for torrent files.  I have found that if I want to save a photo from Facebook on my phone, the easiest way to do it is to "share to Dropbox".

It’s pretty versatile.  I’m working out a security set up that will toss webcam shots into a Dropbox folder as well. 

Why not just use Dropbox?  I think mostly because I really like the UI of Skydrive.  I also trust Microsoft long term a bit more to last… sort of.  I doubt Microsoft will go away anytime soon, but I’m starting to question their viability with the mess that is Windows 8.  Couple in the crazy madness of making Office a subscription service and it feels like a recipe for disaster.  On the other hand, Skydrive is about half the price of Dropbox.  Dropbox is about a dollar per GB for a year, Skydrive is $.50 per GB for a year.  Dropbox’s cheapest plan is also $100 bucks, which kind of sucks.  If they had a 20 GB plan for $25 a year I’d be all over that.  i don’t need 100 GB nor do I want to spend $100 on the service.

What I’m really in the market for is some basic image hosting.  I have a crapload of random images I’ve saved from years of internet that I like having around but I really would like to put them out somewhere for everyone to find and use.  I’ve looked into Imgur and Photobucket a bit but I think both were a bit cost prohibitive.  Dropbox would be great if I could get that non existent 20GB plan, though I don’t think Dropbox gets indexed by search engines at all. 

I’m sure I’ll come up with something.

On a side note, this isn’t a Dropbox ad but here’s my referral link if anyone wants to sign up through it and get me some free space.

Letting Go of Local… Sort Of…

So, baby steps here… Baby steps…

I am more and more making a concerted effort to move myself "to the cloud".  Or more accurately, I’m trying really hard to destroy the death grip that I have on a crapload of "legacy files" and the death grip these files have on me.

Admittedly, I am somewhat inspired by Paul Thurrott and his similar push.  It is something I have wanted to do for a while, and something that needs to be done.  At this point, I operate on three devices, I have family that use two more, I plan to get a Tablet in the near future that will raise this number by one.  I have peripheral family that are getting more online, or in my in law’s case, recently getting a PC for the first time ever.

The point is, I need my data to be more easily accessible and less tied to me and my house.  I talked a bit about one step on this last time, with Carbonite versus Skydrive.  Carbonite is nice, it’s just not super accessible or friendly.  With Skydrive, I can share my photos easily with my parents or my inlaws or whomever I want, yet still keep them private.

I’ve been taking photos since 2006 for the family, I’m halfway through 2006 uploading them.  The hurdle on this starts up soon however as the file size quite literally explodes in 2010 when I got my Pentax K7, the average yearly size has been 2-4Gb, from 2012-2012, it’s 15+Gb.

Which is part of the letting go process.  It’s not as bad in 2012 because I adjusted the settings on my camera to not be quite so huge.  Sure, taking 4000+ by 3000+ photos is great, but do I really need that much resolution?  Sure, occasionally it’s great, but for the most part, 2000-2500 is a good max resolution (on the long side).  Hell anything I take for Tina’s Etsy/Ebay/Facebook has to be shrunk down anyway because at least one of those services only supports 1500 max resolution.

So I’ve been downsizing a lot of these old photos, which significantly reduces the drive space foot print.

I’m also doing something else, something a bit more extreme.  In cases of "non critical" items, I’m flat out deleting the local copies.  I have a folder of photos, taken on my phone, for the Figure of the Day blog that I no longer run.  These photos are also "archived" on Lameazoid.com, so there’s that.  None of them are spectacular, so if I lost them somehow, I am not really out anything.  So I’m putting them up into the Cloud, and deleting the local copy.

It’s not a huge space savings, but when I do this a lot, it adds up.  I’m doing this with a lot of my toy photos.  It’s sort of like… testing the water.  A lot of these are scrub shots anyway where I picked one to crop and use out of 4 or 5, and the "good" ones are on Flickr anyway.  Even if I lost the copies on Flickr and Skydrive, I still own the toy.  On a side note, I’m actually also deleting a lot of the scrub copies and uncropped originals.

The point is, why do I need these?  What am I going to do with these?  If for some reason I really need a "better" iteration than the cropped and adjusted copy I’m using on the blog/Flickr, I can just take a new, better photo.  Keeping this does me no good. 

It’s just… digital baggage. 

Rethinking my Backup Solution

Several years ago, I had a hard drive crap out on me.  I’ve become kind of a Ninja when it comes to data recovery but sometimes there is just nothing that can be done.

I lost some files.  Irritatingly, the hardest hit area was my “personal files” archive of things I have personally created from the past… forever.  Stuff that was somewhat irreplaceable, of course.  It couldn’t have just crushed my Steam Directory or something i could easily download.  Even more irritating is I managed to recover files from the ruins, but they were often corrupted phantom files.  I still come across files I thought I had that no longer load or work.

My wife was, of course, kind of miffed about losing photos.  We had some extra copies of some of them in her Creative Memories photo program, so i could at least recover some of the lost files from that repository, but there were some losses.

So around that time, I signed up for a Carbonite subscription.  Also, no, this is not a Carbonite commercial, kind of the opposite actually.  It took forever but I got all my files uploaded and $60 a year isn’t awful for “peace of mind”.

This was not perfect however.  I have what is best classified as “A shitload” of data.  I did pick and choose a bit, I can redownload some files, or reinstall some programs, but I think when all was said and done I was pushing something like 500 gigs up to Carbonite.  Hey, it’s Unlimited storage so I may as well go nuts.  I did priorities this however.  I added the Photos folder, let it finish, then added more, that sort of thing.

Having all of these files means they are spread out over multiple drives in my computer.  This works fine, Carbonite only rejects network drives (and maybe USB drives).  Originally I had a 1 TB drive split into 3 drive partitions of equal size.  Eventually I outgrew this storage and added a 2 TB drive for a total of 3 TB of storage.  This was where my first problem came in with Carbonite.

I anticipated issues here.  I planned out and took precautions to help alleviate these issues.  My three partitions were E:, F:, and G:.  I disabled Carbonite, stopped it from loading on boot, then installed the new drive.  I split the new drive in half, and merged the old drive back into one volume, copied files around, and assigned the proper drive letters.

Since no one likely managed to follow that, the end result, to the system was, three 333GB drives, E, F, and G; transformed into 3 1TB drives, E, F, and G.  The files were returned to the original corresponding drive paths.  This should have been completely transparent to the computer.  I think it took like a week to shuffle things around.

Restart Carbonite, hook it back up.  Something went wrong.  Essentially, the end result was, I had to completely start my backup anew.  I’m not even real sure what happened to my old data, but now I had to upload hundreds of GB of data again.

It’s probably worth mentioning the second issue I’ve had with Carbonite.  By default, it’s set to run full bore at all times.  This lags not only the internet connection but the machine itself.  It can be scheduled to run only at night and not at all, or as a trickle during the day.  The point is, uploading hundreds of gigs, only during 6 hours in the middle of the night, is a LOOONG process.

Then there is the third issue.  The backup never seems to finish.  I think it’s kind of crapping all over the PC’s multiple user accounts but whatever the case, the backup never seems to finish.  There is always something like 20 gig which never seems to upload, at least on my PC.  It doesn’t help that it only runs when I have my account logged in, and this is a machine that I don’t really use since I built my own machine for my office.

So I’m rethinking this solution.

I really don’t have much that NEEDS to be backed up.  There is a pile of text that I’ve created, blog posts, articles, reviews, code snippets, a handful of photoshopped graphics.  I have my collection of Family photos (which is in fact huge), I have my MP3 collection, I have some video files.

The MP3 solution is done.  I have uploaded all of my MP3s to Google music.  I am not sure I can redownload these, but in a crisis situation, I can at least play through Google Music, which isn’t awful.  I CAN redownload all of my Amazon purchased music.

For my documents, I’ve been slowly adding them to my Google Docs account.  This has an added benefit of making them universally accessible and easily searchable.  I may create a repository of shared documents eventually.

Video is tricky.  Video is huge.  My family videos aren’t too large but I’ve done some freelance video projects that I’ve sold to others and I keep backups of that data.  This solution is a bit more physical, I simply keep copies of the DVDs handy.  If all else fails, I can rip my copy of the DVD and burn fresh copies if someone wants to order an older video.  I probably will never need that 15 gig video file of raw footage, but I like keeping it around.

The most critical item is probably the Photos.  Tina likes them, they are the core of our family memories.  I started adding them to Flickr as a backup but more and more I fear Yahoo may kill Flickr, not necessarily purposefully but possibly simply through attrition.

So I’ve also been trying to use Skydrive as a backup tool.  Microsoft finally gave us a decent interface for using Skydrive and I’ve got 25 gigs at my disposal.  This is actually a problem in my case.  My photo storage, which is family photos from 2005 onward when I first met with my wife Tina, is something like 3 times that amount of space.  I am pretty sure the majority of that is from recent years after I started using the DSLR but it still creates a problem. 

The question is, do I shrink these photos down and reduce the quality?  I think some of it may be HD videos, which I’ll probably just leave to chance.

I actually am considering simply moving my Carbonite money over to paying for Skydrive storage.  Then there is my growing distrust of The Cloud.  Services which get killed or fail.  What happens when the internet is down.  how about when the crushing grip of bandwidth caps makes larger file manipulation infeasible.

This also doesn’t mesh with my desire to make a fresh Linux push wither.  I have not tested it but I’m not sure I can use Skydrive with Ubuntu.

At this point, I’m going to stick with this plan, though I’ll probably do Skydrive AND Flickr.  On the other hand, I’m considering just buying a fireproof safe and shoving an external drive inside of it.

Some Social Archival Tools

So, I am a digital packrat.  It’s only slightly healthier than being a real packrat since it is a “problem” that doesn’t take up a shitload of space or potential harbor disease and dead things.  Hard Drives are cheap, data is cheap to store.  The trick is, keeping it organized.

I’ve tried a couple of different methods over the years for keeping my online resources organized.  That is, my bookmarks.  I have too many bookmarks collected up over years and years of web browsing.  Half of them are dead links, others are simply outdated information.  I’ve come up with two methods recently to keep these bookmarks organized, both of which are easily usable from anywhere, which is kind of key.

Generally speaking, I book mark a website for a handful of reasons.  It’s useful to know these reasons to know how things break down intot he organizational system.

Firstly, there are articles and reference.  News articles on important issues, often political or related to copyright issues, I bookmark these for reference in the future.  The issue with News Articles is that News Outlets seem to not get the idea behind a permalink and either move or delete the article.  also, occasionally, most often in political articles, cover up edits.  So the original becomes a fart in the wind.  This is why I’ve started using Evernote and Evernote Clearly.  Clearly allows you to view an article minus ads and formatting, then lets you clip it to an Evernote file. I was using Instapaper for this function for a while but I am less confident in my ability to quickly and easily retrieve a pile of Instapapers versus Evernote files.

Reference Articles are generally How-Tos.  Step by step articles on how to do little tricks or hacks.

You can find a searchable archive of this data here.

Then there are things that just need links.  Resources of information too large for a single document, links to interesting games or mods for games.  The sort of thing that I just have to hope survive and that others may find useful.

I tried archiving these links in the past using Del.icio.us.  Then there was a scare for a bit where Yahoo was going to close the site.  It survived, I think some group bought it or something, but the unreliability of internet companies makes me want to distrust them with my long term data solutions.

So, I found a skin for WordPress that looks like Delicious and created my own clone.  I just use the basic Press This functionality in WordPress and throw on some tags, instant Bookmark log.

It’s great because I’m hosting it on my own hosting, and I can export an XML backup of the data anytime I want.  Even if WordPress goes away i can still throw up an old version somewhere and import the data back to access it.

Anyway, this repository is located here.