百万数据转换地理编码

导入地址信息文件,调用地理编码信息(高德地理编码)

批量地址数据获取地理编码信息(数据量可达百万)

• 本文章的来自于作者工作中的真实项目,如果看到文章的你有更好的建议,烦请在文章下留言或私信我。

逻辑梳理

1. 获取上传的文件内容,转换为程序可操作的数据;
   1. 地址为空的数据,可以直接保存;
2. 将地址转为需要入库的数据结构;
   1. 地址为数组,当地理编码数据异常时需要将当前数组保存为错误数据(目前是保存到文件中)
   2. 调用批量地理编码接口时,需要注意地址的格式
3. 数据入库。
   1. 正常数据可使用分批入库,节省操作时间;
   2. 为空数据直接入库;
   3. 调用批量地理编码接口查询错误的数据,需要逐一查询确认查询结果无异议后,统一入库。

前言

本文只阐述了作者在开发中的需求和实现,肯定和读者的境况有些不同需要读者自行调整和修改部分代码。对读者有帮助的可能是文件读取、多线程查询、指定大小截取二维数组、多线程入库等部分代码块,文章主要是为有困惑的读者提供部分思路,实际问题需要具体分析和解决。

配置文件

pom.xml

<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
	<groupId>com.alibaba</groupId>
	<artifactId>fastjson</artifactId>
	<version>1.2.47</version>
</dependency>
<dependency>
	<groupId>com.google.code.gson</groupId>
	<artifactId>gson</artifactId>
</dependency>
<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-configuration-processor</artifactId>
	<optional>true</optional>
</dependency>

application.yml

• 数据源配置可以参考作者的另一个篇文章,文章地址SpringBoot配置多数据源

spring:
  profiles:
    active: dev
  servlet:
    multipart:
      enabled: true
      max-file-size: 104857600 # 上传文件大小设置
      max-request-size: 104857600
geo:
  url: https://restapi.amap.com/v3/geocode/geo
  key: # 申请的key

jasypt:
  encryptor:
    password: 484bef877405a9c8

---
# developer environment
spring:
  profiles: dev
  datasource:
    primary:
      driver-class-name: com.cloudera.impala.jdbc41.Driver
      jdbc-url: jdbc:impala://127.0.0.1:21050/default
    secondary:
      driver-class-name: oracle.jdbc.OracleDriver
      # 不加密
      jdbc-url: jdbc:oracle:thin:@//127.0.0.1:1521/testdb
      username: username
      password: pwd12563
fileupload:
  geocode:
    path: # 文件保存地址  D:/fileupload/
    name: # 保存的文件名 data.txt

配置文件实体类

错误数据保存地址

import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
import lombok.Data;
import lombok.ToString;

/**
 * @ClassName: GeocodeFileProperties
 * @Description: 高德地理编码查询失败文件保存地址
 * @author Dew
 * @date 2019/08/09
 */
@Data
@ToString
@Configuration
@ConfigurationProperties(prefix = "fileupload.geocode")
public class GeocodeFileProperties 
	
	/**
	 * 	文件保存路径
	 */
	private String path;
	
	/**
	 * 	文件保存名称
	 */
	private String name;
	
}

高德地理编码配置文件

import lombok.Data;
import lombok.ToString;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;

@Data
@ToString
@Primary
@Configuration
@ConfigurationProperties(prefix = "geo")
public class GeocodingProperties {

	private String url;

	private String key;

}

SpringBoot使用RestTemplate调用RESRful

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.client.ClientHttpRequestFactory;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
import org.springframework.web.client.RestTemplate;

@Configuration
public class RestTemplateConfig {
	
	@Bean
	public RestTemplate restTemplate(ClientHttpRequestFactory factory) {
		return new RestTemplate(factory);
	}
	
	@Bean
	public ClientHttpRequestFactory simplClientHttpRequestFactory() {
		HttpComponentsClientHttpRequestFactory factory = new HttpComponentsClientHttpRequestFactory();
		factory.setConnectionRequestTimeout(10000);
		factory.setConnectTimeout(10000);
		factory.setReadTimeout(10000);
		return factory;
	}
}

实体类

文件内容对应的实体类

import java.io.Serializable;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

/**
 * @ClassName: AddressUserDTO
 * @Description: 用户上传文件对应实体类
 * @date 2019/04/18
 */
@Data
@NoArgsConstructor
@AllArgsConstructor
public class AddressUserDTO implements Serializable {


	/**
	 * 用户AD
	 */
	private String name;

	/**
	 * 用户地址
	 */
	private String address;

	@Override
	public String toString() {
		return "{\"name\":\"" + name + "\", \"address\":\"" + address + "\"}";
	}

}

入库的实体类

import java.io.Serializable;
import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.ToString;

/**
 * @ClassName: GDPoint
 * @Description: 将经纬度转换为 数据库经纬度 50m 栅格编码 :  X= trunc( 经度/0.000129  ), Y= trunc(纬度/0.00245)  取整
 * @date 2019/02/20
 */
@Data
@ToString
@NoArgsConstructor
public class GDPoint implements Serializable {

	private final double X = 0.000129, Y = 0.00245;

	/**
	 * 经度
	 */
	private Integer longitude;

	/**
	 * 维度
	 */
	private Integer latitude;

	public GDPoint(String longitude, String latitude) {
		double lng = Double.parseDouble(longitude);
		double lat = Double.parseDouble(latitude);
		this.longitude = (int) Math.ceil(lng / X);
		this.latitude = (int) Math.ceil(lat / Y);
	}
}

网格数据实体类

import lombok.Data;
import lombok.ToString;

/**
 * @Title 网格数据库实体类映射
 * @Description 数据入库实体类
 * @Author Dew
 * @Date 2019/3/29 15:56
 * @Version 1.0
 */
@Data
@ToString
public class AddressGridDO {
	
	/**
	 * 	地址
	 */
	private String address;
	
	/**
	 * 	经度
	 */
	private String longitude;
	
	/**
	 * 	纬度
	 */
	private String latitude;
	
	/**
	 * 	网格编号X
	 */
	private Integer gridX;
	
	/**
	 *	 网格编号Y
	 */
	private Integer gridY;
	
	public AddressGridDO() {
		
	}
	
	public AddressGridDO(String address) {
		this.address = address;
		this.gridX = 0;
		this.gridY = 0;
		this.longitude = "";
		this.latitude = "";
	}
	
	public AddressGridDO(String address, String longitude, String latitude, Integer gridX, Integer gridY) {
		this.address = address;
		this.longitude = longitude;
		this.latitude = latitude;
		this.gridX = gridX;
		this.gridY = gridY;
	}
	
}

Repository

• 数据库使用的是Oracle,如果是使用MySQL或其他数据库烦请自行百度.暂时没有添加相关代码。

• 数据源配置可以参考作者的另一个篇文章,文章地址SpringBoot配置多数据源

地址转换为网格后的数据入库

import java.util.List;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate;
import org.springframework.jdbc.core.namedparam.SqlParameterSource;
import org.springframework.jdbc.core.namedparam.SqlParameterSourceUtils;
import org.springframework.stereotype.Repository;

@Slf4j
@Repository
public class AddressRepository {

	@Autowired
	private NamedParameterJdbcTemplate secondaryJdbcTemplate;

	/**
	 * @Description 用户网格数据批量入库
	 * @Param [list 入库的数据]
	 * @Date 14:45 2019/4/1
	 * @Return void
	 **/
	public void batchSaveAdvertising(List<AddressGridDO> list) {
		long start = System.currentTimeMillis();

		// merge into 语法是存在则替换,需要特别注意数据更新的条件.以免出现数据被批量更新为同一记录!
		String sql = "merge into t_address_grid t1 using(select :address address,:longitude longitude,:latitude latitude,:gridX gridX,:gridY gridY from dual) t2 on (t1.address = t2.address) when matched then update set t1.longitude=t2.longitude , t1.latitude = t2.latitude , t1.grid_x = t2.gridx , t1.grid_y = t2.gridy when not matched then insert (t1.address,t1.longitude,t1.latitude,t1.grid_x,t1.grid_y) values (t2.address,t2.longitude,t2.latitude,t2.gridx,t2.gridy)";

		SqlParameterSource[] beanSource = SqlParameterSourceUtils.createBatch(list.toArray());
		secondaryJdbcTemplate.batchUpdate(sql, beanSource);

		long end = System.currentTimeMillis();

		log.info("数据入库时间:\t{},入库数据大小:\t{}", (end - start), list.size());
	}
}

Service逻辑实现

import java.io.BufferedInputStream;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.nio.file.FileAlreadyExistsException;
import java.util.ArrayList;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile;

/**
 * @ClassName: GeocodeServiceImpl
 * @Description: 地址信息网格化
 * @date 2019/04/16
 */
@Slf4j
@Service
public class GeocodeServiceImpl implements SynchronizeInformationService {

	private GeocodeFileProperties geocodeFileProperties;

	private AddressRepository addressRepository;

	@Autowired
	public GeocodeServiceImpl(GeocodeFileProperties geocodeFileProperties,
			AddressRepository addressRepository) {
		this.geocodeFileProperties = geocodeFileProperties;
		this.addressRepository = addressRepository;
	}

	@Override
	public Object synchronizationInformation(MultipartFile file) {

		Map<String, Object> result = new HashMap<>();
		result.put("success", false);

		Long timestamp = System.currentTimeMillis();
		try {
			AnalysisText analysisText = new AnalysisText();

			// 读取文件内容,并且拆分为10个一组的list
			Map<String, List<List<AddressUserDTO>>> txtList = analysisText.txt2List(file);
			List<List<AddressUserDTO>> models = txtList.get("group");
			log.info("不为空地址数量:\t" + ((models.size() - 1) * 10 + (models.get(models.size() - 1)).size()));
			this.batchSaveByAddressNotEmpty(models, true, timestamp);

			List<List<AddressUserDTO>> emptys = txtList.get("emptyList");
			log.info("为空地址请求列表数:" + emptys.get(0).size());
			this.batchSaveByAddressNotEmpty(emptys, false, timestamp);

			// 读取查询地理编码错误地址内容,入库
			this.saveErrorData(timestamp);

			result.put("success", true);
			result.put("msg", "导入成功");
		} catch (FileAlreadyExistsException e) {
			log.info("不存在错误地址");
		} catch (IOException e) {
			result.put("msg", "系统错误,请重试或联系管理员");
			e.printStackTrace();
		} catch (Exception e) {
			result.put("msg", "文件内容有误,请联系管理员");
			e.printStackTrace();
		}
		return result;
	}

	/**
	 * @param models txt文本数据
	 * @param flag true: 地址不为空 false: 地址为空 void
	 * @Title: batchSave
	 * @Description: 批量保存地址信息数据,创建的线程数取决于机器的配置。需要根据自身情况增加或减少,如果引起宕机等后果本人概不负责…………
	 */
	private void batchSaveByAddressNotEmpty(List<List<AddressUserDTO>> models, boolean flag, Long timestamp) {
		// 10w 二维数组长度List
		List<List<AddressGridDO>> groupList = new AnalysisText().groupList(models, 3000, flag, timestamp);
		log.info("##查询分组数据大小: {}", groupList.size());
		// 100w 数据待保存,最多创建4个线程
		// 创建线程数 = 数据总数 / groupList
		int threadPoolSize = groupList.size();
		// 最多创建 4 个线程
		threadPoolSize = threadPoolSize >= 4 ? 4 : threadPoolSize;
		log.info("##不为空: {} 地址列表,入库创建的线程数: {}", threadPoolSize);
		ExecutorService executor = Executors.newFixedThreadPool(threadPoolSize);
		try {
			// 拆分网格数据
			for (int i = 0, length = groupList.size(); i < length; i++) {
				List<AddressGridDO> list = groupList.get(i);
				BatchSaveThread saveThread = new BatchSaveThread(list);
				executor.execute(saveThread);
			}
		} catch (Exception e) {
			e.printStackTrace();
			throw new RuntimeException();
		} finally {
			executor.shutdown();
			log.info("##不为空: {},地址列表入库完成", flag);
		}
	}

	/**
	 * @Title: saveErrorData
	 * @Description: 保存查询错误的地址信息 void
	 */
	private void saveErrorData(Long timestamp) throws FileAlreadyExistsException {
		String path = geocodeFileProperties.getPath() + "/" + timestamp + "_" + geocodeFileProperties.getName();
		FileReader reader = null;
		BufferedReader br = null;
		try {
			// 校验文件是否存在
			File file = new File(path);
			if (!file.exists()) {
				log.info("无错误数据");
				throw new FileNotFoundException();
			}

			reader = new FileReader(path);
			br = new BufferedReader(reader);

			// 读取文件内容
			String line;
			List<AddressUserDTO> models = null;
			List<AddressUserDTO> list = new ArrayList<>();
			while ((line = br.readLine()) != null) {
				models = JSON.parseArray(line, AddressUserDTO.class);
				list.addAll(models);
			}
			// 错误数据List
			log.info("错误地址数据量:\t" + list.size());

			List<AddressGridDO> geocode = new ArrayList<>();
			AddressGridDO value = null;

			// 地理编码——经纬度
			String[] array;
			// 用户AD、用户地址、经度,纬度
			String name, address, longitude, latitude;

			GeoCodingUtil geoCodingUtil = new GeoCodingUtil();

			for (AddressUserDTO advertisingModel : list) {
				name = advertisingModel.getName();
				address = advertisingModel.getAddress();

				String point = geoCodingUtil.getShanghaiGeocoding(address);
				if ("[]".equals(point)) {
					value = new AddressGridDO(advertisingModel.getName(), "", "", 0, 0);
					geocode.add(value);
					continue;
				}

				// 获取经纬度
				array = point.split(",");
				longitude = array[0];
				latitude = array[1];

				GDPoint gdPoint = new GDPoint(longitude, latitude);

				value = new AddressGridDO(name, longitude, latitude, gdPoint.getLongitude(), gdPoint.getLatitude());
				geocode.add(value);
			}
			
			// 多线程 执行数据入库
			this.saveBatchErrorData(geocode, 1000);

		} catch (FileNotFoundException fileNotFoundException) {
			log.error("当前导入的文件中没有错误数据");
		} catch (Exception e) {
			log.error("存错误地址信息方法错误,错误内容:\t" + e.getMessage());
			e.printStackTrace();
		} finally {
			try {
				if (reader != null) {
					reader.close();
				}
				if (br != null) {
					br.close();
				}
				// 删除文件
				File file = new File(path);
				if (file.exists() && file.isFile()) {
					file.delete();
					log.info("删除文件路径:\t" + path);
				}
			} catch (IOException e) {
				e.printStackTrace();
			}
		}
	}

	/*
	 * @Title saveBatchErrorData
	 * @Description 批量保存错误地址编码
	 * @Param
	 * @param geocode
	 * @return void
	 **/
	private void saveBatchErrorData(List<AddressGridDO> geocode, int groupSize) {
		// 批量入库
		AnalysisText analysisText = new AnalysisText();
		List<List<AddressGridDO>> geocodeList = analysisText.spilitGroup(geocode, groupSize * 10);

		int threadPoolSize = geocodeList.size();
		threadPoolSize = threadPoolSize >= 4 ? 4 : threadPoolSize;
		ExecutorService executor = Executors.newFixedThreadPool(threadPoolSize);
		try {
			// 拆分网格数据
			for (int i = 0, length = geocodeList.size(); i < length; i++) {
				List<AddressGridDO> list = geocodeList.get(i);
				BatchSaveThread saveThread = new BatchSaveThread(list);
				executor.execute(saveThread);
			}
		} catch (Exception e) {
			e.printStackTrace();
			throw new RuntimeException();
		} finally {
			executor.shutdown();
			log.info("不为空地址列表入库完成");
		}
	}

	/**
	 * @ClassName: BatchSaveThread
	 * @Description: 多线程执行入库操作
	 * @date 2019/03/14
	 */
	class BatchSaveThread implements Runnable {

		private List<AddressGridDO> list;

		public BatchSaveThread(List<AddressGridDO> list) {
			this.list = list;
		}

		@Override
		public void run() {
			if (list.size() > 0) {
				addressRepository.batchSaveAdvertising(list);
			}
		}

	}

	/**
	 * @ClassName: BatchRestAPIThread
	 * @Description: 多线程查询API, 地理编码操作
	 * @date 2019/03/18
	 */
	class BatchRestAPIThread implements Callable<List<AddressGridDO>> {

		AnalysisText analysisText = new AnalysisText();

		private Long timestamp;

		private List<List<AddressUserDTO>> list;

		public BatchRestAPIThread(Long timestamp, List<List<AddressUserDTO>> list) {
			this.timestamp = timestamp;
			this.list = list;
		}

		@Override
		public List<AddressGridDO> call() throws Exception {
			List<AddressGridDO> result = new ArrayList<>();
			for (int i = 0, length = list.size(); i < length; i++) {
				result.addAll(analysisText.geocodes(list.get(i), timestamp));
			}
			return result;
		}

	}

	/**
	 * @ClassName: AnalysisText
	 * @Description: 解析批量导入的txt文件
	 * @date 2019/03/14
	 */
	class AnalysisText {

		/*
		 * @Title txt2List
		 * @Description txt 文件解析为两组list : {"group":地址不为空的List<List<AddressUserDTO>>,"emptyList": address 为空的List}
		 * @Param
		 * @param file
		 * @return java.util.Map<java.lang.String,java.util.List<java.util.List<com.sanss.entity.dto.AddressUserDTO>>>
		 **/
		private Map<String, List<List<AddressUserDTO>>> txt2List(MultipartFile file) throws IOException {
			Map<String, List<List<AddressUserDTO>>> result = new HashMap<>();

			InputStreamReader reader = null;
			BufferedReader br = null;
			List<List<AddressUserDTO>> group = new ArrayList<>();
			List<List<AddressUserDTO>> emptyList = new ArrayList<>();

			AddressUserDTO value = null;
			// 地址不为空的数据
			List<AddressUserDTO> list = new ArrayList<>(10);
			// 地址为空的数据
			List<AddressUserDTO> empty = new ArrayList<>();
			try {

				String encoding = this.getEncoding(file);
				log.info("文件编码格式:\t{}", encoding);

				// 设置字符编码,中文乱码
				// reader = new InputStreamReader(file.getInputStream(), "UTF-8");
				reader = new InputStreamReader(file.getInputStream(), encoding);

				// reader = new InputStreamReader(file.getInputStream(), "GBK");
				// 缓存读取文件
				br = new BufferedReader(reader);
				String line = null;

				// 内数组大小
				int i = 0;
				int j = 0;
				String[] split;
				String userAD, address;
				while ((line = br.readLine()) != null) {
					if (!"".equals(line)) {
						// 设置行内分隔符"，"
						split = line.split(",");
						// 用户AD
						userAD = split[0];
						// 用户地址,示例："世纪大道地铁站***"
						address = split[1];
						address = address.trim().replaceAll("\\*", "");

						// 地址为空与不为空拆分开
						if (!"".equals(address)) {
							value = new AddressUserDTO(userAD, address);
							list.add(value);
							++i;
						} else {
							value = new AddressUserDTO(userAD, "");
							empty.add(value);
							continue;
						}

						// 十个一组分成若干小组
						if (i % 10 == 0 && i > 0) {
							group.add(j, list);
							list = new ArrayList<>(10);
							j++;
						}
					}
				}
				// 行数对10取余不足10的部分
				if (!"[]".equals(list.toString()) && list != null) {
					group.add(j, list);
				}

				// Empty list 不进行 10 分组
				emptyList.add(empty);

				result.put("group", group);
				result.put("emptyList", emptyList);

				return result;
			} catch (IOException e) {
				result.put("group", group);
				result.put("emptyList", emptyList);
				return result;
			} finally {
				reader.close();
				br.close();
				log.info("文件读完:\t" + new Date());
			}
		}

		/*
		 * @Title getEncoding
		 * @Description 获取文件编码
		 * @Param
		 * @param file
		 * @return java.lang.String
		 **/
		private String getEncoding(MultipartFile file) throws IOException {
			BufferedInputStream inputStream = new BufferedInputStream(file.getInputStream());
			int p = (inputStream.read() << 8) + inputStream.read();
			String encoding = null;
			switch (p) {
				case 0xefbb:
					encoding = "UTF-8";
					break;
				case 0xfffe:
					encoding = "Unicode";
					break;
				case 0xfeff:
					encoding = "UTF-16BE";
					break;
				case 0x5c75:
					encoding = "ANSI|ASCII";
					break;
				default:
					encoding = "GBK";
					break;
			}
			return encoding;
		}

		/**
		 * @param groupSize 分组区间
		 * @param flag true:地址不为空|false:地址为空
		 * @Title: groupList
		 * @Description: 地址为空、地址不为空 —— 地理编码
		 */
		private List<List<AddressGridDO>> groupList(List<List<AddressUserDTO>> list, int groupSize, boolean flag,
				Long timestamp) {
			// if (flag) { return this.groupNotEmpty(list, groupSize, timestamp); }
			if (flag) {
				return this.groupNotEmptySinglethread(list, groupSize, timestamp);
			}
			return this.groupEmpty(list, groupSize);
		}

		/**
		 * @param modelGroup 10个地址为一组的List
		 * @param groupSize 分组大小
		 * @param timestamp 访问API接口的时间戳
		 * @Title: groupNotEmptySinglethread
		 * @Description: 不为空地址列表使用单线程查询API, 获取返回结果
		 */
		private List<List<AddressGridDO>> groupNotEmptySinglethread(List<List<AddressUserDTO>> modelGroup, int groupSize,
				Long timestamp) {
			List<AddressGridDO> list = new ArrayList<>();

			List<List<AddressGridDO>> listGroup = new ArrayList<>();
			try {
				AnalysisText analysisText = new AnalysisText();
				for (int i = 0, length = modelGroup.size(); i < length; i++) {
					list.addAll(analysisText.geocodes(modelGroup.get(i), timestamp));
				}

				// 10w 数组查询 API后返回的是100w 数据,数据按 groupSize * 10 一组拆分
				listGroup = this.spilitGroup(list, groupSize * 10);

				return listGroup;
			} catch (Exception e) {
				e.printStackTrace();
				return listGroup;
			}
		}

		/*
		 * @Title groupNotEmptyMultithreading
		 * @Description  groupNotEmptyMultithreading 多线程方法,当前机器使用多线程速率反而更慢
		 *  不为空地址列表使用多线程查询API, 获取返回结果
		 * @Param
		 * @param modelGroup 10个地址为一组的List
		 * @param groupSize 分组大小
		 * @param timestamp 访问API接口的时间戳
		 * @return java.util.List<java.util.List<com.sanss.entity.AddressGridDO>>
		 **/
		@Deprecated
		private List<List<AddressGridDO>> groupNotEmptyMultithreading(List<List<AddressUserDTO>> modelGroup, int groupSize,
				Long timestamp) {
			List<AddressGridDO> list = new ArrayList<>();

			List<List<AddressGridDO>> listGroup = new ArrayList<>();
			ExecutorService executor = null;
			try {
				// List 长度
				int listSize = modelGroup.size();
				int runSize = (listSize / groupSize) + 1;

				List<List<AddressUserDTO>> value = null;
				List<List<List<AddressUserDTO>>> models = new ArrayList<>();
				for (int i = 0; i < runSize; i++) {
					int start = i * groupSize;
					if (i + 1 == runSize) {
						int end = listSize;
						value = modelGroup.subList(start, end);
					} else {
						int end = (i + 1) * groupSize;
						value = modelGroup.subList(start, end);
					}
					models.add(value);
				}

				// 创建多线程查询API
				int threadPoolSize = models.size();
				executor = Executors.newFixedThreadPool(threadPoolSize);
				threadPoolSize = threadPoolSize >= 4 ? 4 : threadPoolSize;

				log.info("查询API线程数:\t" + threadPoolSize);

				for (int i = 0, length = models.size(); i < length; i++) {
					BatchRestAPIThread apiThread = new BatchRestAPIThread(timestamp, models.get(i));
					Future<List<AddressGridDO>> values = executor.submit(apiThread);
					list.addAll(values.get());
				}

				// 10w 数组查询 API后返回的是100w 数据,数据按1000一组拆分
				listGroup = this.spilitGroup(list, groupSize * 10);

				return listGroup;
			} catch (Exception e) {
				e.printStackTrace();
				return listGroup;
			} finally {
				if (executor != null) {
					executor.shutdown();
				}
			}
		}

		/**
		 * @Title: groupEmpty
		 * @Description: 地址为空的数据拆分
		 */
		private List<List<AddressGridDO>> groupEmpty(List<List<AddressUserDTO>> modelList, int groupSize) {
			List<AddressGridDO> adList = new ArrayList<>();
			// 数组下标为0
			List<AddressUserDTO> models = modelList.get(0);
			AddressGridDO adUserGridDO = null;
			String name;
			for (int i = 0; i < models.size(); i++) {
				AddressUserDTO model = models.get(i);
				name = model.getName();
				adUserGridDO = new AddressGridDO(name, "", "", 0, 0);
				adList.add(adUserGridDO);
			}
			return this.spilitGroup(adList, groupSize * 10);
		}

		/**
		 * @param values List 数组
		 * @param groupSize 数组分组大小
		 * @Title: spilitGroup
		 * @Description: 数据分成 groupSize 大的数组,多线程执行入库
		 */
		private List<List<AddressGridDO>> spilitGroup(List<AddressGridDO> values, int groupSize) {
			List<List<AddressGridDO>> listGroup = new ArrayList<>();
			// List 长度
			int listSize = values.size();
			int runSize = (listSize / groupSize) + 1;

			List<AddressGridDO> value = null;
			for (int i = 0; i < runSize; i++) {
				int start = i * groupSize;
				if (i + 1 == runSize) {
					int end = listSize;
					value = values.subList(start, end);
				} else {
					int end = (i + 1) * groupSize;
					value = values.subList(start, end);
				}
				listGroup.add(value);
			}
			return listGroup;
		}

		/**
		 * @return List<AddressGridDO>
		 * @Title: geocodes
		 * @Description: 将地址转为 网格,只取address不为空的数据
		 */
		private List<AddressGridDO> geocodes(List<AddressUserDTO> adUserDto, Long timestamp) {
			List<AddressGridDO> result = new ArrayList<>();

			try {
				// 高德地理编码,批量请求地址参数
				List<String> address = new ArrayList<>();
				for (int i = 0; i < adUserDto.size(); i++) {
					address.add(adUserDto.get(i).getAddress());
				}
				// 查询地理编码 —— 上海
				List<String> points = new GeoCodingUtil().getShanghaiGeocoding(address);

				// 查询的地址个数返回结果不等于地址个数时,抛出异常
				if (points.size() != adUserDto.size()) {
					/// log.error("查询地址个数不等于返回个数:\t" + JSON.toJSONString(adUserDto) + "\t返回结果:\t" + points);
					throw new Exception();
				}

				AddressGridDO value = null;

				// 地理编码——经纬度
				String[] array;
				// 经度,纬度
				String longitude, latitude;
				// 用户编号
				String name;

				for (int i = 0; i < points.size(); i++) {
					name = adUserDto.get(i).getName();

					/** 将地理编码转换为50m栅格编码*/
					String point = points.get(i);

					// point:[[],[121.389822,31.258292]]
					if ("[]".equals(point)) {
						value = new AddressGridDO(name, "", "", 0, 0);
						result.add(value);
						continue;
					}

					// 获取经纬度
					array = point.split(",");
					longitude = array[0];
					latitude = array[1];

					GDPoint gdPoint = new GDPoint(longitude, latitude);

					value = new AddressGridDO(name, longitude, latitude, gdPoint.getLongitude(), gdPoint.getLatitude());
					result.add(value);
				}
				return result;
			} /*
				catch (SocketTimeoutException e) {
				this.appendDataFile(adUserDto, timestamp);
				} catch (NullPointerException e) {
				this.appendDataFile(adUserDto, timestamp);
				} 
				*/ catch (Exception e) {
				this.appendDataFile(adUserDto, timestamp);
			}
			return result;
		}

		/**
		 * @param models void
		 * @Title: createDataFile
		 * @Description: 查询异常的用户地址保存为文件
		 */
		private void appendDataFile(List<AddressUserDTO> models, Long timestamp) {
			StringBuilder sb = new StringBuilder();
			// models.toString() 方法是返回JSON字符串,并不是普通的toString()方法

			sb.append(JSON.toJSONString(models) + System.getProperty("line.separator"));
			FileWriter fw = null;
			PrintWriter pw = null;
			String txtName = timestamp + "_" + geocodeFileProperties.getName();
			String txtPath = geocodeFileProperties.getPath();

			try {
				File file = new File(txtPath);
				if (!file.exists()) {
					file.mkdirs();
				}

				fw = new FileWriter(file + "/" + txtName, true);
				pw = new PrintWriter(fw);

				pw.print(sb.toString());
				pw.flush();
				fw.flush();

			} catch (Exception e) {
				e.printStackTrace();
			} finally {
				try {
					pw.close();
					fw.close();
				} catch (IOException e) {
					e.printStackTrace();
				}
			}
		}
	}

}

高德地理编码API工具类

二维数组拆分为指定大小

import java.util.ArrayList;
import java.util.List;

/**
 * @ClassName ArrayUtil
 * @Description <br/> 数组操作工具类
 * @Author Dew
 * @Date 2019/4/1 13:53
 * @Version 1.0
 **/
public class ArrayUtil<T> {

	/**
	 * @Author Dew
	 * @Description 拆分二维数组为指定大小
	 * @Param [values 待分组Array 分组, groupSize 分组大小]
	 * @Date 13:54 2019/4/1
	 * @Return java.util.List<T>
	 **/
	public List<List<T>> spilitGroup(List<T> values, int groupSize) {
		List<List<T>> listGroup = new ArrayList<>();
		// List 长度
		int listSize = values.size();
		int runSize = (listSize / groupSize) + 1;

		List<T> value = null;
		for (int i = 0; i < runSize; i++) {
			int start = i * groupSize;
			if (i + 1 == runSize) {
				int end = listSize;
				value = values.subList(start, end);
			} else {
				int end = (i + 1) * groupSize;
				value = values.subList(start, end);
			}
			listGroup.add(value);
		}
		return listGroup;
	}

}

高德地理编码API调用实例

• 注: 在非 controller 中读取配置文件时获取不到配置类的属性值,欲了解详情可以看下我的另一篇博客SpringBoot配置文件详解

import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import javax.annotation.PostConstruct;
import lombok.Data;
import lombok.ToString;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestTemplate;

/**
 * @ClassName: GeoDto
 * @Description: 调用高德地理编码API传递的参数实体类
 * @date 2019/03/27
 */
@Data
class GeoDto {

	private Boolean batch;

	private String city;

	private String address;

	private String url;

	private String key;

	public GeoDto() {
	}

	public GeoDto(List<String> address, String city, Boolean batch, String url, String key) {
		StringBuilder addressSb = new StringBuilder();
		for (int i = 0, length = address.size(); i < length; i++) {
			addressSb.append(address.get(i));
			addressSb.append("|");
		}
		this.address = addressSb.substring(0, addressSb.lastIndexOf("|")).toString();
		this.city = city;
		this.batch = batch;
		this.url = url;
		this.key = key;
	}

	@Override
	public String toString() {
		return url + "?batch=" + batch + "&key=" + key + "&city=" + city + "&address=" + address;
	}

}

/**
 * @ClassName: Geocodes
 * @Description: 返回结果实体类, 只获取了location:经纬度 API 地址：https://lbs.amap.com/api/webservice/guide/api/georegeo 返回结果示例:
 * {"status":"1","info":"OK","infocode":"10000","count":"1","geocodes":[{"formatted_address":"上海市浦东新区东方明珠","country":"中国","province":"上海市","citycode":"021","city":"上海市","district":"浦东新区","township":[],"neighborhood":{"name":[],"type":[]},"building":{"name":[],"type":[]},"adcode":"310115","street":[],"number":[],"location":"121.499740,31.239853","level":"兴趣点"}]}
 * @date 2019/03/27
 */

@Data
@ToString
class Geocodes {

	private String location;

}

/**
 * @Title
 * @Description 高德地理编码API返回JSON实体类, 部分需要的属性, 完整的返回值在上方
 * @return
 **/
@Data
@ToString
class Geocode {

	private int status;

	private int count;

	private List<Geocodes> geocodes;

}

/**
 * @ClassName: GeoCodingUtil
 * @Description: 高德地理编码/逆编码(将地址转换为经度,纬度) API: https://lbs.amap.com/api/webservice/guide/api/georegeo/#scene
 * @date 2019/03/14
 */
@Slf4j
@Component
public class GeoCodingUtil {

	@Autowired
	private RestTemplate restTemplate;

	@Autowired
	private GeocodingProperties properties;

	private static GeoCodingUtil geoCodingUtil;

	private static Long index = 0L;

	@PostConstruct
	private void init() {
		geoCodingUtil = this;
		geoCodingUtil.properties = this.properties;
		geoCodingUtil.restTemplate = this.restTemplate;

		log.info("请求API地址:\t{}", properties.getUrl());
	}

	/**
	 * @param address 地址List
	 * @param city 城市名称,例如:上海,北京
	 * @Title: getGeocoding
	 * @Description: 获取批量地址地理编码
	 * @Return List<String>
	 */
	public List<String> getGeocoding(List<String> address, String city) throws Exception {
		return getGeocoding(address, city, true);
	}

	/**
	 * @param address 地址List
	 * @Title: getShanghaiGeocoding
	 * @Description: 获取上海地区批量地址的地理编码
	 * @ReturnType: List<String>
	 */
	public List<String> getShanghaiGeocoding(List<String> address) throws Exception {
		return getGeocoding(address, "上海", true);
	}

	/**
	 * @param address 地址
	 * @Title: getShanghaiGeocoding
	 * @Description: 获取上海地区单个地址的地理编码
	 * @ReturnType: String
	 */
	public String getShanghaiGeocoding(String address) throws Exception {
		List<String> list = new ArrayList<String>();
		list.add(address);
		list = getGeocoding(list, "上海", false);

		String empty = new ArrayList<>().toString();
		if (empty.equals(list.toString())) {
			return list.toString();
		}
		return list.get(0);
	}

	/**
	 * @param address 最多支持十个地址拼接
	 * @param city 城市
	 * @param batch 是否批量
	 * @Title: getGeocoding
	 * @Description: 上海地区的高德地理编码获取
	 * @Return List<String>
	 */
	private List<String> getGeocoding(List<String> address, String city, boolean batch) throws Exception {
		List<String> point = new ArrayList<>();

		GeoDto geoDto = new GeoDto(address, city, batch, geoCodingUtil.properties.getUrl(),
				geoCodingUtil.properties.getKey());
		String realUrl = geoDto.toString();

		Geocode geocode = geoCodingUtil.restTemplate.getForObject(realUrl, Geocode.class);

		int count = geocode.getCount();
		if (count > 0) {
			List<Geocodes> geocodes = geocode.getGeocodes();
			Iterator<Geocodes> iterator = geocodes.iterator();
			String location;
			while (iterator.hasNext()) {
				location = iterator.next().getLocation();
				if (location == null) {
					point.add("");
					continue;
				}
				point.add(location);
			}
		}
		return point;
	}
}

常见问题

RestTemplate 调用API超时

解决方案

1. 配置类RestTemplateConfig中不设置connectTimeout、readTimeout
2. 将超时时间变长,timeout时间单位为 毫秒

实现多线程有返回值的执行

/**
	 * @ClassName: BatchRestAPIThread
	 * @Description: 多线程查询API,地理编码操作
	 * @date 2019/03/18
	 */
	class BatchRestAPIThread implements Callable<List<InfoDo>> {

		private List<List<InfoDto>> list;

		public BatchRestAPIThread(List<List<InfoDto>> list) {
			this.list = list;
		}

		@Override
		public List<InfoDo> call() throws Exception {
			List<InfoDo> result = new ArrayList<>();
			// 执行代码块
			return result;
		}

	}