MyPage is a personalized page based on your interests.The page is customized to help you to find content that matters you the most.


I'm not curious

Sqoop vs Lingual

0
Started on: 21 August 14
Participants: 1
I'm facing a scenario that I could use some experience-based advice on. My client is a large bank that needs to import some risk transaction data into Hadoop. We're currently looking at Sqoop and Lingual (Cascading) and trying to make a decision.

Sqoop or Lingual capabilities aside, the inhouse developers seem to be very comfortable with SQL because of having worked a lot on BI, and somewhat less so with Java. That gives me a bit of a starting bias in favour of Lingual, but my concern there is that the extra layer of interpretation going into providing a fuller SQL interface may impact performance negatively. The import needs to be done on a fairly frequent basis, and the data volumes to be migrated are in the order of about 15 TB each time. The data is all alphanumeric, with no images or files.

We'll probably have to run prototypes sooner or later to find out, but I was interested in hearing from anyone who has already run into a similar situation.



I'm facing a scenario that I could use some experience-based advice on. My client is a large bank that needs to import some risk transaction data into Hadoop. We're currently looking at Sqoop and Lingual (Cascading) and trying to make a decision.

Sqoop or Lingual capabilities aside, the inhouse developers seem to be very comfortable with SQL because of having worked a lot on BI, and somewhat less so with Java. That gives me a bit of a starting bias in favour of Lingual, but my concern there is that the extra layer of interpretation going into providing a fuller SQL interface may impact performance negatively. The import needs to be done on a fairly frequent basis, and the data volumes to be migrated are in the order of about 15 TB each time. The data is all alphanumeric, with no images or files.

We'll probably have to run prototypes sooner or later to find out, but I was interested in hearing from anyone who has already run into a similar situation.

This challenge is listed under Open Source , Development & Implementations and Data & Information Management Community

Related Posts:

Initiator

Mario
Mario
  Follow

Business technology consultant

More posts by: Mario Lewis

1 Suggestion

  1. 26 August 14
    0

    Lingual provides a full ANSI SQL interface for Hadoop, thus allowing for easier integration with BI tools and helping SQL skilled personnel to use Hadoop immediately. It also includes a SQL shell. The queries are executed in batch nevrtheless. The data movement into Hadoop is quicker.

    Sqoop works by running a query on the relational database and exporting the resulting rows into files in either one of these formats: text, binary, Avro, or Sequence Files. These files can then be saved on Hadoop’s HDFS.Sqoops imported files can be viewed through Hive or Pig.

    Given the above and the available inhouse expertise, you may think about going with Lingual. Note the data volume.

Awards & Accolades for MyTechLogy
Winner of
REDHERRING
Top 100 Asia
Finalist at SiTF Awards 2014 under the category Best Social & Community Product
Finalist at HR Vendor of the Year 2015 Awards under the category Best Learning Management System
Finalist at HR Vendor of the Year 2015 Awards under the category Best Talent Management Software
Hidden Image Url

Back to Top