hadoop mapreduce多表关联 假设有如下两个文件,一个是表是公司和地址的序号的对应,一个表是地址的序号和地址的名称的对应。 表1: [plain] A:Beijing Red Star 1 A:Shenzhen Thunder 3 A:Guangzhou Honda 2 A:Beijing Rising 1 A:Guangzhou Development Ba hadoop mapreduce多表关联假设有如下两个文件,一个是表是公司和地址的序号的对应,一个表是地址的序号和地址的名称的对应。
表1:
[plain]A:Beijing Red Star 1 A:Shenzhen Thunder 3 A:Guangzhou Honda 2 A:Beijing Rising 1 A:Guangzhou Development Bank 2 A:Tencent 3 A:Back of Beijing 1
表2:
[plain]B:1 Beijing B:2 Guangzhou B:3 Shenzhen B:4 Xian
mapreduce如下:
[plain]private static final Text typeA = new Text("A:");private static final Text typeB = new Text("B:");private static Log log = LogFactory.getLog(MTJoin.class);public static class Map extends Mapper{public void map(Object key, Text value, Context context) throws IOException, InterruptedException {String valueStr = value.toString();String type = valueStr.substring(0, 2);String content = valueStr.substring(2);log.info(content);if(type.equals("A:")){String[] contentArray = content.split("\t");String city = contentArray[0];String address = contentArray[1];MapWritable map = new MapWritable();map.put(typeA, new Text(city));context.write(new Text(address), map);}else if(type.equals("B:")){String[] contentArray = content.split("\t");String adrNum = contentArray[0];String adrName = contentArray[1];MapWritable map = new MapWritable();map.put(typeB, new Text(adrName));context.write(new Text(adrNum), map);}}}public static class Reduce extends Reducer{public void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException {Iterator it = values.iterator();List cityList = new ArrayList();List adrList = new ArrayList();while(it.hasNext()){MapWritable map = it.next();if(map.containsKey(typeA)){cityList.add((Text)map.get(typeA));}else if(map.containsKey(typeB)){adrList.add((Text)map.get(typeB));}}for(int i = 0; i{for(int j = 0; j{context.write(cityList.get(i), adrList.get(j));}}}}原理很简单,map的出口,以地址的序号作为key,然后出来的时候,公司名称放一个list,地址的名称放一个list,两个list的内容作笛卡儿积,就得到了结果。
输出如下:
[plain]Beijing Red Star Beijing Beijing Rising Beijing Back of Beijing Beijing Guangzhou Honda Guangzhou Guangzhou Development Bank Guangzhou Shenzhen Thunder Shenzhen Tencent Shenzhen
