训练集构造 - Biendata

Discussion

Home Discussion Board baifendian2019 训练集构造

LightR

训练集构造

posted in baifendian2019

2019-11-08 05:58

需要先把`train_set.xml`文件打开，把首行的`utf8`改成`utf-8`，不然python没法解析
```python
#coding=utf-8 
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
from xml.dom.minidom import parse

def generate_train_data_pair(equ_questions, not_equ_questions):
    a = [x+"\t"+y+"\t"+"0" for x in equ_questions for y in not_equ_questions]
    b = [x+"\t"+y+"\t"+"1" for x in equ_questions for y in equ_questions if x!=y]
    return a+b

def parse_train_data(xml_data):
    pair_list = []
    doc = parse(xml_data)
    collection = doc.documentElement
    for i in collection.getElementsByTagName("Questions"):
        # if i.hasAttribute("number"):
        #     print ("Questions number=", i.getAttribute("number"))
        EquivalenceQuestions = i.getElementsByTagName("EquivalenceQuestions")
        NotEquivalenceQuestions = i.getElementsByTagName("NotEquivalenceQuestions")
        equ_questions = EquivalenceQuestions[0].getElementsByTagName("question")
        not_equ_questions = NotEquivalenceQuestions[0].getElementsByTagName("question")
        equ_questions_list, not_equ_questions_list = [], [] 
        for q in equ_questions:
            try:
                equ_questions_list.append(q.childNodes[0].data.strip())
            except:
                continue
        for q in not_equ_questions:
            try:
                not_equ_questions_list.append(q.childNodes[0].data.strip())
            except:
                continue
        pair = generate_train_data_pair(equ_questions_list, not_equ_questions_list)
        pair_list.extend(pair)
    print("All pair count=", len(pair_list))
    return pair_list

def write_train_data(file, pairs):
    with open(file, "w") as f:
        for pair in pairs:
            f.write(pair+"\n")

if __name__ == "__main__":
    pair_list = parse_train_data("./baifendian_data/train_set.xml")
    write_train_data("./baifendian_data/train_data.txt", pair_list)
```

23 comments

广告位招租

2019-11-11 01:52

claireting

2019-11-11 06:54

nothing

2019-11-12 14:57

LightR reply nothing

2019-11-13 15:17

Reply

0

<p>每个人生成训练集的规则不同，看个人爱好</p>

HYQ

2019-11-15 08:20

xiaolu

2019-11-19 11:36