Parts of Speech (POS) and Tokenizing
Start by find a poem from the Poetry Foundation. Something new from their website is audio recordings.
A nice R package that can be used to tag the parts of speech of words in a document is RDRPOSTagger. Note that this package depends on Java, so it might not work for you.
# install.packages("rJava")
# install.packages("data.table")
# devtools::install_github("bnosac/RDRPOSTagger")library(RDRPOSTagger)Loading required package: rJava
Registered S3 method overwritten by 'data.table':
method from
print.data.table models <- rdr_available_models()
modelsRiple down Rule based available taggers:
----------------------------------------
1/ POS tagging for languages:
English, French, German, Hindi, Italian, Thai, Vietnamese
2/ MORPH tagging for languages:
Bulgarian, Czech, Dutch, French, German, Portuguese, Spanish, Swedish
3/ UniversalPOS tagging for languages:
Ancient_Greek-PROIEL, Ancient_Greek, Arabic, Basque, Belarusian, Bulgarian, Catalan, Chinese, Coptic, Croatian, Czech-CAC, Czech-CLTT, Czech, Danish, Dutch-LassySmall, Dutch, English-LinES, English-ParTUT, English, Estonian, Finnish-FTB, Finnish, French-ParTUT, French-Sequoia, French, Galician-TreeGal, Galician, German, Gothic, Greek, Hebrew, Hindi, Hungarian, Indonesian, Irish, Italian-ParTUT, Italian, Japanese, Korean, Latin-ITTB, Latin-PROIEL, Latin, Latvian, Lithuanian, Norwegian-Bokmaal, Norwegian-Nynorsk, Old_Church_Slavonic, Persian, Polish, Portuguese-BR, Portuguese, Romanian, Russian-SynTagRus, Russian, Slovak, Slovenian-SST, Slovenian, Spanish-AnCora, Spanish, Swedish-LinES, Swedish, Tamil, Turkish, Urdu, Vietnamesemodels$MORPH$language[1] "Bulgarian" "Czech" "Dutch" "French" "German"
[6] "Portuguese" "Spanish" "Swedish" models$POS$language[1] "English" "French" "German" "Hindi" "Italian"
[6] "Thai" "Vietnamese"models$UniversalPOS$language [1] "Ancient_Greek-PROIEL" "Ancient_Greek"
[3] "Arabic" "Basque"
[5] "Belarusian" "Bulgarian"
[7] "Catalan" "Chinese"
[9] "Coptic" "Croatian"
[11] "Czech-CAC" "Czech-CLTT"
[13] "Czech" "Danish"
[15] "Dutch-LassySmall" "Dutch"
[17] "English-LinES" "English-ParTUT"
[19] "English" "Estonian"
[21] "Finnish-FTB" "Finnish"
[23] "French-ParTUT" "French-Sequoia"
[25] "French" "Galician-TreeGal"
[27] "Galician" "German"
[29] "Gothic" "Greek"
[31] "Hebrew" "Hindi"
[33] "Hungarian" "Indonesian"
[35] "Irish" "Italian-ParTUT"
[37] "Italian" "Japanese"
[39] "Korean" "Latin-ITTB"
[41] "Latin-PROIEL" "Latin"
[43] "Latvian" "Lithuanian"
[45] "Norwegian-Bokmaal" "Norwegian-Nynorsk"
[47] "Old_Church_Slavonic" "Persian"
[49] "Polish" "Portuguese-BR"
[51] "Portuguese" "Romanian"
[53] "Russian-SynTagRus" "Russian"
[55] "Slovak" "Slovenian-SST"
[57] "Slovenian" "Spanish-AnCora"
[59] "Spanish" "Swedish-LinES"
[61] "Swedish" "Tamil"
[63] "Turkish" "Urdu"
[65] "Vietnamese" An example from the github.
x <- c("Oleg Borisovich Kulik is a Ukrainian-born Russian performance artist")
tagger <- rdr_model(language = "English", annotation = "UniversalPOS")
rdr_pos(tagger, x = x)Type in your own sentences.
x <- c("Have a nice summer. Hope you will continue your study of text processing.")
tagger <- rdr_model(language = "English", annotation = "UniversalPOS")
rdr_pos(tagger, x = x)For a poem examine the distribution of the parts of speech.
Use the poem May BY JONATHAN GALASSI
library(tidyverse)── Attaching core tidyverse packages ──────────── tidyverse 2.0.0 ──
✔ dplyr 1.1.1 ✔ readr 2.1.4
✔ forcats 1.0.0 ✔ stringr 1.5.0
✔ ggplot2 3.4.1 ✔ tibble 3.2.1
✔ lubridate 1.9.2 ✔ tidyr 1.3.0
✔ purrr 1.0.1 ── Conflicts ────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the ]8;;http://conflicted.r-lib.org/conflicted package]8;; to force all conflicts to become errorsx <- c("May
BY JONATHAN GALASSI
The backyard apple tree gets sad so soon,
takes on a used-up, feather-duster look
within a week.
The ivy’s spring reconnaissance campaign
sends red feelers out and up and down
to find the sun.
Ivy from last summer clogs the pool,
brewing a loamy, wormy, tea-leaf mulch
soft to the touch
and rank with interface of rut and rot.
The month after the month they say is cruel
is and is not.")
tagger <- rdr_model(language = "English", annotation = "UniversalPOS")
tagger_df <- as_tibble(rdr_pos(tagger, x = x))
tagger_df
tagger_df %>%
count(pos, sort = TRUE)
tagger_df %>%
count(pos, sort = TRUE) %>%
mutate(pos = reorder(pos, n)) %>%
ggplot(aes(pos, n)) +
geom_col() +
xlab(NULL) +
coord_flip()
library(wordcloud)Loading required package: RColorBrewertagger_df %>%
count(pos) %>%
with(wordcloud(pos, n))
There are many R packages for text mining. We have looked at sentimentr in the past. Check out tokenizer.
Use the poem To My Mother BY EDGAR ALLAN POE
library(tokenizers)
x <- c("To My Mother
BY EDGAR ALLAN POE
Because I feel that, in the Heavens above,
The angels, whispering to one another,
Can find, among their burning terms of love,
None so devotional as that of “Mother,”
Therefore by that dear name I long have called you—
You who are more than mother unto me,
And fill my heart of hearts, where Death installed you
In setting my Virginia's spirit free.
My mother—my own mother, who died early,
Was but the mother of myself; but you
Are mother to the one I loved so dearly,
And thus are dearer than the mother I knew
By that infinity with which my wife
Was dearer to my soul than its soul-life.
")
tokenize_characters(x)[[1]] [1] "t" "o" "m" "y" "m" "o" "t" "h" "e" "r" "b" "y" "e" "d" "g"
[16] "a" "r" "a" "l" "l" "a" "n" "p" "o" "e" "b" "e" "c" "a" "u"
[31] "s" "e" "i" "f" "e" "e" "l" "t" "h" "a" "t" "i" "n" "t" "h"
[46] "e" "h" "e" "a" "v" "e" "n" "s" "a" "b" "o" "v" "e" "t" "h"
[61] "e" "a" "n" "g" "e" "l" "s" "w" "h" "i" "s" "p" "e" "r" "i"
[76] "n" "g" "t" "o" "o" "n" "e" "a" "n" "o" "t" "h" "e" "r" "c"
[91] "a" "n" "f" "i" "n" "d" "a" "m" "o" "n" "g" "t" "h" "e" "i"
[106] "r" "b" "u" "r" "n" "i" "n" "g" "t" "e" "r" "m" "s" "o" "f"
[121] "l" "o" "v" "e" "n" "o" "n" "e" "s" "o" "d" "e" "v" "o" "t"
[136] "i" "o" "n" "a" "l" "a" "s" "t" "h" "a" "t" "o" "f" "m" "o"
[151] "t" "h" "e" "r" "t" "h" "e" "r" "e" "f" "o" "r" "e" "b" "y"
[166] "t" "h" "a" "t" "d" "e" "a" "r" "n" "a" "m" "e" "i" "l" "o"
[181] "n" "g" "h" "a" "v" "e" "c" "a" "l" "l" "e" "d" "y" "o" "u"
[196] "y" "o" "u" "w" "h" "o" "a" "r" "e" "m" "o" "r" "e" "t" "h"
[211] "a" "n" "m" "o" "t" "h" "e" "r" "u" "n" "t" "o" "m" "e" "a"
[226] "n" "d" "f" "i" "l" "l" "m" "y" "h" "e" "a" "r" "t" "o" "f"
[241] "h" "e" "a" "r" "t" "s" "w" "h" "e" "r" "e" "d" "e" "a" "t"
[256] "h" "i" "n" "s" "t" "a" "l" "l" "e" "d" "y" "o" "u" "i" "n"
[271] "s" "e" "t" "t" "i" "n" "g" "m" "y" "v" "i" "r" "g" "i" "n"
[286] "i" "a" "s" "s" "p" "i" "r" "i" "t" "f" "r" "e" "e" "m" "y"
[301] "m" "o" "t" "h" "e" "r" "m" "y" "o" "w" "n" "m" "o" "t" "h"
[316] "e" "r" "w" "h" "o" "d" "i" "e" "d" "e" "a" "r" "l" "y" "w"
[331] "a" "s" "b" "u" "t" "t" "h" "e" "m" "o" "t" "h" "e" "r" "o"
[346] "f" "m" "y" "s" "e" "l" "f" "b" "u" "t" "y" "o" "u" "a" "r"
[361] "e" "m" "o" "t" "h" "e" "r" "t" "o" "t" "h" "e" "o" "n" "e"
[376] "i" "l" "o" "v" "e" "d" "s" "o" "d" "e" "a" "r" "l" "y" "a"
[391] "n" "d" "t" "h" "u" "s" "a" "r" "e" "d" "e" "a" "r" "e" "r"
[406] "t" "h" "a" "n" "t" "h" "e" "m" "o" "t" "h" "e" "r" "i" "k"
[421] "n" "e" "w" "b" "y" "t" "h" "a" "t" "i" "n" "f" "i" "n" "i"
[436] "t" "y" "w" "i" "t" "h" "w" "h" "i" "c" "h" "m" "y" "w" "i"
[451] "f" "e" "w" "a" "s" "d" "e" "a" "r" "e" "r" "t" "o" "m" "y"
[466] "s" "o" "u" "l" "t" "h" "a" "n" "i" "t" "s" "s" "o" "u" "l"
[481] "l" "i" "f" "e"tokenize_words(x)[[1]]
[1] "to" "my" "mother" "by"
[5] "edgar" "allan" "poe" "because"
[9] "i" "feel" "that" "in"
[13] "the" "heavens" "above" "the"
[17] "angels" "whispering" "to" "one"
[21] "another" "can" "find" "among"
[25] "their" "burning" "terms" "of"
[29] "love" "none" "so" "devotional"
[33] "as" "that" "of" "mother"
[37] "therefore" "by" "that" "dear"
[41] "name" "i" "long" "have"
[45] "called" "you" "you" "who"
[49] "are" "more" "than" "mother"
[53] "unto" "me" "and" "fill"
[57] "my" "heart" "of" "hearts"
[61] "where" "death" "installed" "you"
[65] "in" "setting" "my" "virginia's"
[69] "spirit" "free" "my" "mother"
[73] "my" "own" "mother" "who"
[77] "died" "early" "was" "but"
[81] "the" "mother" "of" "myself"
[85] "but" "you" "are" "mother"
[89] "to" "the" "one" "i"
[93] "loved" "so" "dearly" "and"
[97] "thus" "are" "dearer" "than"
[101] "the" "mother" "i" "knew"
[105] "by" "that" "infinity" "with"
[109] "which" "my" "wife" "was"
[113] "dearer" "to" "my" "soul"
[117] "than" "its" "soul" "life" tokenize_word_stems(x)[[1]]
[1] "to" "my" "mother" "by" "edgar"
[6] "allan" "poe" "becaus" "i" "feel"
[11] "that" "in" "the" "heaven" "abov"
[16] "the" "angel" "whisper" "to" "one"
[21] "anoth" "can" "find" "among" "their"
[26] "burn" "term" "of" "love" "none"
[31] "so" "devot" "as" "that" "of"
[36] "mother" "therefor" "by" "that" "dear"
[41] "name" "i" "long" "have" "call"
[46] "you" "you" "who" "are" "more"
[51] "than" "mother" "unto" "me" "and"
[56] "fill" "my" "heart" "of" "heart"
[61] "where" "death" "instal" "you" "in"
[66] "set" "my" "virginia" "spirit" "free"
[71] "my" "mother" "my" "own" "mother"
[76] "who" "die" "earli" "was" "but"
[81] "the" "mother" "of" "myself" "but"
[86] "you" "are" "mother" "to" "the"
[91] "one" "i" "love" "so" "dear"
[96] "and" "thus" "are" "dearer" "than"
[101] "the" "mother" "i" "knew" "by"
[106] "that" "infin" "with" "which" "my"
[111] "wife" "was" "dearer" "to" "my"
[116] "soul" "than" "it" "soul" "life" library(stopwords)
tokenize_words(x, stopwords = stopwords::stopwords("en"))[[1]]
[1] "mother" "edgar" "allan" "poe"
[5] "feel" "heavens" "angels" "whispering"
[9] "one" "another" "can" "find"
[13] "among" "burning" "terms" "love"
[17] "none" "devotional" "mother" "therefore"
[21] "dear" "name" "long" "called"
[25] "mother" "unto" "fill" "heart"
[29] "hearts" "death" "installed" "setting"
[33] "virginia's" "spirit" "free" "mother"
[37] "mother" "died" "early" "mother"
[41] "mother" "one" "loved" "dearly"
[45] "thus" "dearer" "mother" "knew"
[49] "infinity" "wife" "dearer" "soul"
[53] "soul" "life" The Penn Treebank tokenizer preserves punctuation and separates common English contractions.
tokenize_ptb(x)[[1]]
[1] "To" "My" "Mother" "BY"
[5] "EDGAR" "ALLAN" "POE" "Because"
[9] "I" "feel" "that" ","
[13] "in" "the" "Heavens" "above"
[17] "," "The" "angels" ","
[21] "whispering" "to" "one" "another"
[25] "," "Can" "find" ","
[29] "among" "their" "burning" "terms"
[33] "of" "love" "," "None"
[37] "so" "devotional" "as" "that"
[41] "of" "“Mother" "," "”"
[45] "Therefore" "by" "that" "dear"
[49] "name" "I" "long" "have"
[53] "called" "you—" "You" "who"
[57] "are" "more" "than" "mother"
[61] "unto" "me" "," "And"
[65] "fill" "my" "heart" "of"
[69] "hearts" "," "where" "Death"
[73] "installed" "you" "In" "setting"
[77] "my" "Virginia" "'s" "spirit"
[81] "free." "My" "mother—my" "own"
[85] "mother" "," "who" "died"
[89] "early" "," "Was" "but"
[93] "the" "mother" "of" "myself"
[97] ";" "but" "you" "Are"
[101] "mother" "to" "the" "one"
[105] "I" "loved" "so" "dearly"
[109] "," "And" "thus" "are"
[113] "dearer" "than" "the" "mother"
[117] "I" "knew" "By" "that"
[121] "infinity" "with" "which" "my"
[125] "wife" "Was" "dearer" "to"
[129] "my" "soul" "than" "its"
[133] "soul-life" "." tokenize_ngrams(x, n = 5, n_min = 2,
stopwords = stopwords::stopwords("en"))[[1]]
[1] "mother edgar"
[2] "mother edgar allan"
[3] "mother edgar allan poe"
[4] "mother edgar allan poe feel"
[5] "edgar allan"
[6] "edgar allan poe"
[7] "edgar allan poe feel"
[8] "edgar allan poe feel heavens"
[9] "allan poe"
[10] "allan poe feel"
[11] "allan poe feel heavens"
[12] "allan poe feel heavens angels"
[13] "poe feel"
[14] "poe feel heavens"
[15] "poe feel heavens angels"
[16] "poe feel heavens angels whispering"
[17] "feel heavens"
[18] "feel heavens angels"
[19] "feel heavens angels whispering"
[20] "feel heavens angels whispering one"
[21] "heavens angels"
[22] "heavens angels whispering"
[23] "heavens angels whispering one"
[24] "heavens angels whispering one another"
[25] "angels whispering"
[26] "angels whispering one"
[27] "angels whispering one another"
[28] "angels whispering one another can"
[29] "whispering one"
[30] "whispering one another"
[31] "whispering one another can"
[32] "whispering one another can find"
[33] "one another"
[34] "one another can"
[35] "one another can find"
[36] "one another can find among"
[37] "another can"
[38] "another can find"
[39] "another can find among"
[40] "another can find among burning"
[41] "can find"
[42] "can find among"
[43] "can find among burning"
[44] "can find among burning terms"
[45] "find among"
[46] "find among burning"
[47] "find among burning terms"
[48] "find among burning terms love"
[49] "among burning"
[50] "among burning terms"
[51] "among burning terms love"
[52] "among burning terms love none"
[53] "burning terms"
[54] "burning terms love"
[55] "burning terms love none"
[56] "burning terms love none devotional"
[57] "terms love"
[58] "terms love none"
[59] "terms love none devotional"
[60] "terms love none devotional mother"
[61] "love none"
[62] "love none devotional"
[63] "love none devotional mother"
[64] "love none devotional mother therefore"
[65] "none devotional"
[66] "none devotional mother"
[67] "none devotional mother therefore"
[68] "none devotional mother therefore dear"
[69] "devotional mother"
[70] "devotional mother therefore"
[71] "devotional mother therefore dear"
[72] "devotional mother therefore dear name"
[73] "mother therefore"
[74] "mother therefore dear"
[75] "mother therefore dear name"
[76] "mother therefore dear name long"
[77] "therefore dear"
[78] "therefore dear name"
[79] "therefore dear name long"
[80] "therefore dear name long called"
[81] "dear name"
[82] "dear name long"
[83] "dear name long called"
[84] "dear name long called mother"
[85] "name long"
[86] "name long called"
[87] "name long called mother"
[88] "name long called mother unto"
[89] "long called"
[90] "long called mother"
[91] "long called mother unto"
[92] "long called mother unto fill"
[93] "called mother"
[94] "called mother unto"
[95] "called mother unto fill"
[96] "called mother unto fill heart"
[97] "mother unto"
[98] "mother unto fill"
[99] "mother unto fill heart"
[100] "mother unto fill heart hearts"
[101] "unto fill"
[102] "unto fill heart"
[103] "unto fill heart hearts"
[104] "unto fill heart hearts death"
[105] "fill heart"
[106] "fill heart hearts"
[107] "fill heart hearts death"
[108] "fill heart hearts death installed"
[109] "heart hearts"
[110] "heart hearts death"
[111] "heart hearts death installed"
[112] "heart hearts death installed setting"
[113] "hearts death"
[114] "hearts death installed"
[115] "hearts death installed setting"
[116] "hearts death installed setting virginia's"
[117] "death installed"
[118] "death installed setting"
[119] "death installed setting virginia's"
[120] "death installed setting virginia's spirit"
[121] "installed setting"
[122] "installed setting virginia's"
[123] "installed setting virginia's spirit"
[124] "installed setting virginia's spirit free"
[125] "setting virginia's"
[126] "setting virginia's spirit"
[127] "setting virginia's spirit free"
[128] "setting virginia's spirit free mother"
[129] "virginia's spirit"
[130] "virginia's spirit free"
[131] "virginia's spirit free mother"
[132] "virginia's spirit free mother mother"
[133] "spirit free"
[134] "spirit free mother"
[135] "spirit free mother mother"
[136] "spirit free mother mother died"
[137] "free mother"
[138] "free mother mother"
[139] "free mother mother died"
[140] "free mother mother died early"
[141] "mother mother"
[142] "mother mother died"
[143] "mother mother died early"
[144] "mother mother died early mother"
[145] "mother died"
[146] "mother died early"
[147] "mother died early mother"
[148] "mother died early mother mother"
[149] "died early"
[150] "died early mother"
[151] "died early mother mother"
[152] "died early mother mother one"
[153] "early mother"
[154] "early mother mother"
[155] "early mother mother one"
[156] "early mother mother one loved"
[157] "mother mother"
[158] "mother mother one"
[159] "mother mother one loved"
[160] "mother mother one loved dearly"
[161] "mother one"
[162] "mother one loved"
[163] "mother one loved dearly"
[164] "mother one loved dearly thus"
[165] "one loved"
[166] "one loved dearly"
[167] "one loved dearly thus"
[168] "one loved dearly thus dearer"
[169] "loved dearly"
[170] "loved dearly thus"
[171] "loved dearly thus dearer"
[172] "loved dearly thus dearer mother"
[173] "dearly thus"
[174] "dearly thus dearer"
[175] "dearly thus dearer mother"
[176] "dearly thus dearer mother knew"
[177] "thus dearer"
[178] "thus dearer mother"
[179] "thus dearer mother knew"
[180] "thus dearer mother knew infinity"
[181] "dearer mother"
[182] "dearer mother knew"
[183] "dearer mother knew infinity"
[184] "dearer mother knew infinity wife"
[185] "mother knew"
[186] "mother knew infinity"
[187] "mother knew infinity wife"
[188] "mother knew infinity wife dearer"
[189] "knew infinity"
[190] "knew infinity wife"
[191] "knew infinity wife dearer"
[192] "knew infinity wife dearer soul"
[193] "infinity wife"
[194] "infinity wife dearer"
[195] "infinity wife dearer soul"
[196] "infinity wife dearer soul soul"
[197] "wife dearer"
[198] "wife dearer soul"
[199] "wife dearer soul soul"
[200] "wife dearer soul soul life"
[201] "dearer soul"
[202] "dearer soul soul"
[203] "dearer soul soul life"
[204] "soul soul"
[205] "soul soul life"
[206] "soul life" tokenize_sentences(x) [[1]]
[1] "To My Mother BY EDGAR ALLAN POE Because I feel that, in the Heavens above, The angels, whispering to one another, Can find, among their burning terms of love, None so devotional as that of “Mother,” Therefore by that dear name I long have called you— You who are more than mother unto me, And fill my heart of hearts, where Death installed you In setting my Virginia's spirit free."
[2] "My mother—my own mother, who died early, Was but the mother of myself; but you Are mother to the one I loved so dearly, And thus are dearer than the mother I knew By that infinity with which my wife Was dearer to my soul than its soul-life." tokenize_paragraphs(x)[[1]]
[1] "To My Mother BY EDGAR ALLAN POE"
[2] "Because I feel that, in the Heavens above, The angels, whispering to one another, Can find, among their burning terms of love, None so devotional as that of “Mother,” Therefore by that dear name I long have called you— You who are more than mother unto me, And fill my heart of hearts, where Death installed you In setting my Virginia's spirit free. My mother—my own mother, who died early, Was but the mother of myself; but you Are mother to the one I loved so dearly, And thus are dearer than the mother I knew By that infinity with which my wife Was dearer to my soul than its soul-life. "LS0tCnRpdGxlOiAiUGFydHMgb2YgU3BlZWNoIChQT1MpIGFuZCBUb2tlbml6aW5nIgpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sKLS0tCgpTdGFydCBieSBmaW5kIGEgcG9lbSBmcm9tIHRoZSBbUG9ldHJ5IEZvdW5kYXRpb25dKGh0dHBzOi8vd3d3LnBvZXRyeWZvdW5kYXRpb24ub3JnLykuICBTb21ldGhpbmcgbmV3IGZyb20gdGhlaXIgd2Vic2l0ZSBpcyBhdWRpbyByZWNvcmRpbmdzLgoKQSBuaWNlIFIgcGFja2FnZSB0aGF0IGNhbiBiZSB1c2VkIHRvIHRhZyB0aGUgcGFydHMgb2Ygc3BlZWNoIG9mIHdvcmRzIGluIGEgZG9jdW1lbnQgaXMgW1JEUlBPU1RhZ2dlcl0oaHR0cHM6Ly9naXRodWIuY29tL2Jub3NhYy9SRFJQT1NUYWdnZXIpLiAgTm90ZSB0aGF0IHRoaXMgcGFja2FnZSBkZXBlbmRzIG9uIEphdmEsIHNvIGl0IG1pZ2h0IG5vdCB3b3JrIGZvciB5b3UuCgpgYGB7cn0KIyBpbnN0YWxsLnBhY2thZ2VzKCJySmF2YSIpCiMgaW5zdGFsbC5wYWNrYWdlcygiZGF0YS50YWJsZSIpCiMgZGV2dG9vbHM6Omluc3RhbGxfZ2l0aHViKCJibm9zYWMvUkRSUE9TVGFnZ2VyIikKYGBgCgpgYGB7cn0KbGlicmFyeShSRFJQT1NUYWdnZXIpCm1vZGVscyA8LSByZHJfYXZhaWxhYmxlX21vZGVscygpCgptb2RlbHMKCm1vZGVscyRNT1JQSCRsYW5ndWFnZQptb2RlbHMkUE9TJGxhbmd1YWdlCm1vZGVscyRVbml2ZXJzYWxQT1MkbGFuZ3VhZ2UKYGBgCgpBbiBleGFtcGxlIGZyb20gdGhlIGdpdGh1Yi4KCmBgYHtyfQp4IDwtIGMoIk9sZWcgQm9yaXNvdmljaCBLdWxpayBpcyBhIFVrcmFpbmlhbi1ib3JuIFJ1c3NpYW4gcGVyZm9ybWFuY2UgYXJ0aXN0IikKdGFnZ2VyIDwtIHJkcl9tb2RlbChsYW5ndWFnZSA9ICJFbmdsaXNoIiwgYW5ub3RhdGlvbiA9ICJVbml2ZXJzYWxQT1MiKQpyZHJfcG9zKHRhZ2dlciwgeCA9IHgpCmBgYAoKVHlwZSBpbiB5b3VyIG93biBzZW50ZW5jZXMuCgpgYGB7cn0KeCA8LSBjKCJIYXZlIGEgbmljZSBzdW1tZXIuICBIb3BlIHlvdSB3aWxsIGNvbnRpbnVlIHlvdXIgc3R1ZHkgb2YgdGV4dCBwcm9jZXNzaW5nLiIpCnRhZ2dlciA8LSByZHJfbW9kZWwobGFuZ3VhZ2UgPSAiRW5nbGlzaCIsIGFubm90YXRpb24gPSAiVW5pdmVyc2FsUE9TIikKcmRyX3Bvcyh0YWdnZXIsIHggPSB4KQpgYGAKCkZvciBhIHBvZW0gZXhhbWluZSB0aGUgZGlzdHJpYnV0aW9uIG9mIHRoZSBwYXJ0cyBvZiBzcGVlY2guCgpVc2UgdGhlIHBvZW0gW01heQpCWSBKT05BVEhBTiBHQUxBU1NJXShodHRwczovL3d3dy5wb2V0cnlmb3VuZGF0aW9uLm9yZy9wb2Vtcy80Njg4OC9tYXktNTZkMjI2ZjhlOTg2OCkKCmBgYHtyfQpsaWJyYXJ5KHRpZHl2ZXJzZSkKCnggPC0gYygiTWF5CkJZIEpPTkFUSEFOIEdBTEFTU0kKClRoZSBiYWNreWFyZCBhcHBsZSB0cmVlIGdldHMgc2FkIHNvIHNvb24sCnRha2VzIG9uIGEgdXNlZC11cCwgZmVhdGhlci1kdXN0ZXIgbG9vawp3aXRoaW4gYSB3ZWVrLgoKVGhlIGl2eeKAmXMgc3ByaW5nIHJlY29ubmFpc3NhbmNlIGNhbXBhaWduCnNlbmRzIHJlZCBmZWVsZXJzIG91dCBhbmQgdXAgYW5kIGRvd24KdG8gZmluZCB0aGUgc3VuLgoKSXZ5IGZyb20gbGFzdCBzdW1tZXIgY2xvZ3MgdGhlIHBvb2wsCmJyZXdpbmcgYSBsb2FteSwgd29ybXksIHRlYS1sZWFmIG11bGNoCnNvZnQgdG8gdGhlIHRvdWNoCgphbmQgcmFuayB3aXRoIGludGVyZmFjZSBvZiBydXQgYW5kIHJvdC4KVGhlIG1vbnRoIGFmdGVyIHRoZSBtb250aCB0aGV5IHNheSBpcyBjcnVlbAppcyBhbmQgaXMgbm90LiIpCgp0YWdnZXIgPC0gcmRyX21vZGVsKGxhbmd1YWdlID0gIkVuZ2xpc2giLCBhbm5vdGF0aW9uID0gIlVuaXZlcnNhbFBPUyIpCnRhZ2dlcl9kZiA8LSBhc190aWJibGUocmRyX3Bvcyh0YWdnZXIsIHggPSB4KSkKdGFnZ2VyX2RmCgp0YWdnZXJfZGYgJT4lIAogIGNvdW50KHBvcywgc29ydCA9IFRSVUUpIAoKdGFnZ2VyX2RmICU+JSAKICBjb3VudChwb3MsIHNvcnQgPSBUUlVFKSAlPiUKICAgbXV0YXRlKHBvcyA9IHJlb3JkZXIocG9zLCBuKSkgJT4lCiAgZ2dwbG90KGFlcyhwb3MsIG4pKSArCiAgZ2VvbV9jb2woKSArCiAgeGxhYihOVUxMKSArCiAgY29vcmRfZmxpcCgpCgpsaWJyYXJ5KHdvcmRjbG91ZCkKCnRhZ2dlcl9kZiAlPiUKICBjb3VudChwb3MpICU+JQogIHdpdGgod29yZGNsb3VkKHBvcywgbikpCmBgYAoKVGhlcmUgYXJlIG1hbnkgUiBwYWNrYWdlcyBmb3IgdGV4dCBtaW5pbmcuICBXZSBoYXZlIGxvb2tlZCBhdCBzZW50aW1lbnRyIGluIHRoZSBwYXN0LiAgQ2hlY2sgb3V0IFt0b2tlbml6ZXJdKGh0dHBzOi8vY3Jhbi5yLXByb2plY3Qub3JnL3dlYi9wYWNrYWdlcy90b2tlbml6ZXJzL3ZpZ25ldHRlcy9pbnRyb2R1Y3Rpb24tdG8tdG9rZW5pemVycy5odG1sKS4KClVzZSB0aGUgcG9lbSBbVG8gTXkgTW90aGVyIEJZIEVER0FSIEFMTEFOIFBPRV0oaHR0cHM6Ly93d3cucG9ldHJ5Zm91bmRhdGlvbi5vcmcvcG9lbXMvNTU1NjkvdG8tbXktbW90aGVyLTU2ZDIzNzRjMGQ0MzQpCgoKYGBge3J9CmxpYnJhcnkodG9rZW5pemVycykKCnggPC0gYygiVG8gTXkgTW90aGVyCkJZIEVER0FSIEFMTEFOIFBPRQoKQmVjYXVzZSBJIGZlZWwgdGhhdCwgaW4gdGhlIEhlYXZlbnMgYWJvdmUsClRoZSBhbmdlbHMsIHdoaXNwZXJpbmcgdG8gb25lIGFub3RoZXIsCkNhbiBmaW5kLCBhbW9uZyB0aGVpciBidXJuaW5nIHRlcm1zIG9mIGxvdmUsCk5vbmUgc28gZGV2b3Rpb25hbCBhcyB0aGF0IG9mIOKAnE1vdGhlcizigJ0KVGhlcmVmb3JlIGJ5IHRoYXQgZGVhciBuYW1lIEkgbG9uZyBoYXZlIGNhbGxlZCB5b3XigJQKWW91IHdobyBhcmUgbW9yZSB0aGFuIG1vdGhlciB1bnRvIG1lLApBbmQgZmlsbCBteSBoZWFydCBvZiBoZWFydHMsIHdoZXJlIERlYXRoIGluc3RhbGxlZCB5b3UKSW4gc2V0dGluZyBteSBWaXJnaW5pYSdzIHNwaXJpdCBmcmVlLgpNeSBtb3RoZXLigJRteSBvd24gbW90aGVyLCB3aG8gZGllZCBlYXJseSwKV2FzIGJ1dCB0aGUgbW90aGVyIG9mIG15c2VsZjsgYnV0IHlvdQpBcmUgbW90aGVyIHRvIHRoZSBvbmUgSSBsb3ZlZCBzbyBkZWFybHksCkFuZCB0aHVzIGFyZSBkZWFyZXIgdGhhbiB0aGUgbW90aGVyIEkga25ldwpCeSB0aGF0IGluZmluaXR5IHdpdGggd2hpY2ggbXkgd2lmZQpXYXMgZGVhcmVyIHRvIG15IHNvdWwgdGhhbiBpdHMgc291bC1saWZlLgoiKQoKdG9rZW5pemVfY2hhcmFjdGVycyh4KVtbMV1dIAoKdG9rZW5pemVfd29yZHMoeCkKCnRva2VuaXplX3dvcmRfc3RlbXMoeCkKCmxpYnJhcnkoc3RvcHdvcmRzKQp0b2tlbml6ZV93b3Jkcyh4LCBzdG9wd29yZHMgPSBzdG9wd29yZHM6OnN0b3B3b3JkcygiZW4iKSkKYGBgCgpUaGUgUGVubiBUcmVlYmFuayB0b2tlbml6ZXIgcHJlc2VydmVzIHB1bmN0dWF0aW9uIGFuZCBzZXBhcmF0ZXMgY29tbW9uIEVuZ2xpc2ggY29udHJhY3Rpb25zLgoKYGBge3J9CnRva2VuaXplX3B0Yih4KQpgYGAKCmBgYHtyfQp0b2tlbml6ZV9uZ3JhbXMoeCwgbiA9IDUsIG5fbWluID0gMiwKICAgICAgICAgICAgICAgIHN0b3B3b3JkcyA9IHN0b3B3b3Jkczo6c3RvcHdvcmRzKCJlbiIpKQpgYGAKCgpgYGB7cn0KdG9rZW5pemVfc2VudGVuY2VzKHgpIApgYGAKCgpgYGB7cn0KdG9rZW5pemVfcGFyYWdyYXBocyh4KQpgYGAKCg==