Python torchtext.data.interleave_keys() Examples

The following are 12 code examples of torchtext.data.interleave_keys(). You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may also want to check out all available functions/classes of the module torchtext.data , or try the search function

Example #1

Source File: semantic_similar_data.py From glyce with Apache License 2.0

6 votes

def __init__(self, args):
        self.RAW = data.RawField()
        self.RAW.is_target = False
        tokenize = lambda x: list(x)
        self.TEXT = data.Field(batch_first=True, tokenize=tokenize)
        self.LABEL = data.Field(sequential=False, unk_token=None)
        self.train, self.dev, self.test = data.TabularDataset.splits(
            path='/data/nfsdata/nlp/datasets/sentence_pair/bq_corpus_torch10',
            train='BQ_train.json',
            validation='BQ_dev.json',
            test='BQ_test.json',
            format='json',
            fields={"gold_label": ("label", self.LABEL),
                    "sentence1": ("q1", self.TEXT),
                    "sentence2": ("q2", self.TEXT),
                    "ID": ("id", self.RAW)})

        self.TEXT.build_vocab(self.train, self.dev, self.test, vectors=Vectors("BQ300", args.data))
        self.LABEL.build_vocab(self.train)

        sort_key = lambda x: data.interleave_keys(len(x.q1), len(x.q2))
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        self.train_iter = data.BucketIterator(self.train, batch_size=args.batch_size, device=device, sort_key=sort_key, sort=True)
        self.dev_iter = data.BucketIterator(self.dev, batch_size=args.batch_size, device=device, sort_key=sort_key, sort=True)
        self.test_iter = data.BucketIterator(self.test, batch_size=args.batch_size, device=device, sort_key=sort_key, sort=True)

Example #2

Source File: qe_dataset.py From OpenKiwi with GNU Affero General Public License v3.0

5 votes

def sort_key(ex):
        # don't work for pack_padded_sequences
        # return data.interleave_keys(len(ex.source), len(ex.target))
        return len(ex.source)

Example #3

Source File: data.py From dl4mt-nonauto with BSD 3-Clause "New" or "Revised" License

5 votes

def sort_key(ex):
        return data.interleave_keys(len(ex.src), len(ex.trg))

Example #4

Source File: translation.py From transformer-pytorch with MIT License

5 votes

def sort_key(ex):
        return data.interleave_keys(len(ex.src), len(ex.trg))

Example #5

Source File: text.py From interpretable_predictions with MIT License

5 votes

def sort_key(ex):
        return data.interleave_keys(
            len(ex.premise), len(ex.hypothesis))

Example #6