Lub tswv yim ntawm cov ntaub ntawv xov xwm entropy cuam tshuam qhov tsis zoo logarithm ntawm qhov tshwm sim loj ua haujlwm rau tus nqi. Yog li, thaum cov ntaub ntawv qhov chaw muaj tus nqi nrog qhov tsawg dua qhov tshwm sim (piv txwv li, thaum muaj xwm txheej nrog qhov tshwm sim tsis tshua muaj tshwm sim), qhov xwm txheej muaj ntau "cov ntaub ntawv" ("xav tsis thoob") dua li thaum cov ntaub ntawv muaj txiaj ntsig nrog qhov tshwm sim ntau dua..
Tus nqi ntawm cov ntaub ntawv xa tawm los ntawm txhua qhov xwm txheej tau teev tseg hauv txoj kev no dhau los ua qhov sib txawv tsis sib xws uas nws xav tau tus nqi yog cov ntaub ntawv entropy. Feem ntau, entropy hais txog kev tsis sib haum xeeb lossis tsis paub tseeb, thiab nws cov ntsiab lus siv hauv cov ntaub ntawv kev tshawb xav yog ncaj qha piv rau qhov uas siv nyob rau hauv statistical thermodynamics. Lub tswv yim ntawm IE tau qhia los ntawm Claude Shannon hauv nws daim ntawv xyoo 1948 "A Mathematical Theory of Communication". Nov yog qhov lo lus "Shannon's informational entropy" los ntawm.
Definition and system
Tus qauv yooj yim ntawm cov ntaub ntawv xa tawm muaj peb lub ntsiab lus: cov ntaub ntawv qhov chaw, kev sib txuas lus channel thiab tus txais,thiab, raws li Shannon tso nws, "qhov teeb meem kev sib txuas lus yooj yim" yog rau tus neeg txais yuav tuaj yeem txheeb xyuas cov ntaub ntawv dab tsi tau tsim los ntawm lub hauv paus raws li lub teeb liab nws tau txais los ntawm cov channel. Entropy muab qhov txwv tsis pub tshaj ntawm qhov luv tshaj qhov nruab nrab tsis muaj qhov encoding ntev ntawm cov ntaub ntawv compressed. Yog tias qhov entropy ntawm qhov chaw tsawg dua qhov bandwidth ntawm kev sib txuas lus channel, cov ntaub ntawv nws tsim tuaj yeem xa mus rau cov neeg txais kev ntseeg siab (tsawg kawg hauv txoj kev xav, tej zaum tsis quav ntsej txog qee qhov kev xav xws li qhov nyuaj ntawm qhov system yuav tsum tau xa cov ntaub ntawv. thiab lub sijhawm nws yuav siv sijhawm los xa cov ntaub ntawv).
Cov ntaub ntawv entropy feem ntau ntsuas hauv cov khoom (xws li hu ua "shannons") lossis qee zaum hauv "natural units" (nats) lossis decimal qhov chaw (hu ua "dits", "bans" lossis "hartleys"). Chav ntsuas ntsuas nyob ntawm lub hauv paus ntawm lub logarithm, uas yog siv los txiav txim qhov entropy.
Properties thiab logarithm
Lub cav qhov tshwm sim ntawm kev faib tawm yog qhov tseem ceeb raws li kev ntsuas ntawm entropy vim nws yog additive rau qhov chaw ywj pheej. Piv txwv li, lub entropy ntawm ib tug ncaj ncees thawj koom ruam ntawm ib npib yog 1 ntsis, thaum lub entropy ntawm m-ntim yog m ntsis. Nyob rau hauv ib qho yooj yim sawv cev, log2(n) cov khoom yuav tsum tau los sawv cev rau ib tug txawv txav uas yuav coj mus rau ib qho ntawm n qhov tseem ceeb yog n yog ib lub hwj chim ntawm 2. Yog hais tias cov nuj nqis no sib npaug, qhov entropy (hauv cov khoom) yog sib npaug rau tus lej ntawd. Yog tias ib qho ntawm cov txiaj ntsig zoo dua li lwm tus, qhov kev soj ntsuam uas nws yoglub ntsiab lus tshwm sim, yog cov ntaub ntawv tsawg dua yog tias qee qhov tshwm sim tsawg dua yuav tshwm sim. Hloov pauv, cov xwm txheej tsis tshua muaj muab cov ntaub ntawv taug qab ntxiv.
Vim tias qhov kev soj ntsuam ntawm cov xwm txheej tsis tshua muaj tshwm sim tsawg dua, tsis muaj ib yam dab tsi uas zoo sib xws tias qhov entropy (xav tias yog cov ntaub ntawv nruab nrab) tau los ntawm cov ntaub ntawv tsis sib xws yog ib txwm tsawg dua lossis sib npaug rau log2(n). Entropy yog xoom thaum ib qho txiaj ntsig tau txhais.
Shannon cov ntaub ntawv entropy ntsuas qhov kev txiav txim siab no thaum paub txog qhov tshwm sim ntawm cov ntaub ntawv hauv qab. Lub ntsiab lus ntawm kev soj ntsuam cov xwm txheej (lub ntsiab lus ntawm cov lus) tsis cuam tshuam rau hauv lub ntsiab lus ntawm entropy. Cov yav tas yuav siv sij hawm mus rau hauv tus account tsuas yog qhov tshwm sim ntawm pom ib qho kev tshwm sim, yog li cov ntaub ntawv nws encapsulates yog cov ntaub ntawv hais txog lub hauv paus tis ntawm possibilities, tsis yog hais txog lub ntsiab lus ntawm cov xwm txheej lawv tus kheej. Cov khoom ntawm cov ntaub ntawv entropy tseem zoo ib yam li tau piav qhia saum toj no.
Information theory
Lub tswv yim tseem ceeb ntawm cov ntaub ntawv kev tshawb xav yog qhov ntau tus paub txog ib lub ncauj lus, cov ntaub ntawv tsawg dua ib tus tuaj yeem tau txais txog nws. Yog tias qhov xwm txheej yuav tshwm sim, nws tsis yog qhov xav tsis thoob thaum nws tshwm sim thiab yog li muab cov ntaub ntawv tshiab me ntsis. Hloov chaw, yog tias qhov xwm txheej tsis tuaj yeem tshwm sim, nws tau qhia ntau ntxiv tias qhov xwm txheej tshwm sim. Yog li ntawd, lub payload yog ib qho kev nce ntxiv ntawm qhov tshwm sim ntawm qhov tshwm sim (1 / p).
Tam sim no yog tias muaj xwm txheej tshwm sim, entropyntsuas qhov nruab nrab cov ntaub ntawv cov ntsiab lus koj tuaj yeem xav tau yog tias muaj ib qho xwm txheej tshwm sim. Qhov no txhais tau tias casting tuag muaj entropy ntau dua li pov ib npib vim hais tias txhua qhov kev tshwm sim siv lead ua muaj qhov tsawg dua qhov tshwm sim ntawm txhua qhov txiaj ntsig.
Ntse
Yog li, entropy yog ib qho kev ntsuas ntawm qhov tsis paub tseeb ntawm lub xeev lossis, uas yog tib yam, nws cov ntsiab lus nruab nrab ntawm cov ntaub ntawv. Kom tau txais kev nkag siab zoo ntawm cov ntsiab lus no, xav txog qhov piv txwv ntawm kev xaiv nom tswv. Feem ntau cov kev xaiv tsa no tshwm sim vim qhov tshwm sim ntawm, piv txwv li, kev xaiv tsa tseem tsis tau paub.
Hauv lwm lo lus, cov txiaj ntsig ntawm kev tshawb fawb yog qhov tsis tuaj yeem kwv yees, thiab qhov tseeb, ua nws thiab tshuaj xyuas cov ntaub ntawv muab qee cov ntaub ntawv tshiab; lawv tsuas yog txoj kev sib txawv ntawm qhov hais tias qhov ua ntej entropy ntawm qhov kev xaiv tsa yog loj.
Tam sim no xav txog qhov xwm txheej uas tib qhov kev xaiv tsa tau ua thib ob sai tom qab thawj zaug. Txij li thaum cov txiaj ntsig ntawm thawj daim ntawv ntsuam xyuas twb paub lawm, cov txiaj ntsig ntawm qhov kev tshawb fawb thib ob tuaj yeem kwv yees tau zoo thiab cov txiaj ntsig yuav tsum tsis muaj ntau cov ntaub ntawv tshiab; Hauv qhov no, qhov tseem ceeb ntawm qhov kev xaiv tsa thib ob yog qhov tsawg dua piv rau thawj qhov.
Npaj Tos
Tam sim no xav txog qhov piv txwv ntawm flipping ib npib. Piv txwv tias qhov tshwm sim ntawm tails yog tib yam li qhov tshwm sim ntawm lub taub hau, qhov entropy ntawm lub npib tos yog siab heev, vim nws yog ib qho piv txwv tshwj xeeb ntawm cov ntaub ntawv xov xwm ntawm lub system.
Qhov no yog vimtias nws tsis tuaj yeem kwv yees tias qhov txiaj ntsig ntawm ib lub npib raug pov tseg ua ntej: yog tias peb yuav tsum xaiv, qhov zoo tshaj plaws peb tuaj yeem ua tau yog kwv yees tias cov nyiaj npib yuav tsaws ntawm tails, thiab qhov kev kwv yees no yuav raug nrog qhov tshwm sim ntawm 1 / 2. Xws li ib lub npib pov muaj ib qho me ntsis entropy, vim muaj ob qhov tshwm sim uas tshwm sim nrog qhov sib npaug, thiab kawm txog qhov tshwm sim tiag tiag muaj cov ntaub ntawv me ntsis.
Ntawm qhov tsis sib xws, flipping ib lub npib siv ob sab nrog tails thiab tsis muaj lub taub hau muaj xoom entropy vim lub npib yuav ib txwm tsaws ntawm lub cim no thiab qhov tshwm sim tuaj yeem kwv yees zoo kawg nkaus.
Zoo kawg
Yog tias lub tswv yim compression tsis poob, txhais tau tias koj tuaj yeem rov qab tau tag nrho cov lus qub los ntawm kev decompressing, ces cov lus compressed muaj cov ntaub ntawv zoo ib yam li tus thawj, tab sis kis tau tsawg dua cov cim. Ntawd yog, nws muaj cov ntaub ntawv ntau dua lossis siab dua entropy ib tus cwj pwm. Qhov no txhais tau hais tias cov lus compressed muaj tsawg dua redundancy.
Roughly hais, Shannon lub hauv paus code coding theorem hais tias ib tug lossless compression scheme tsis tuaj yeem txo cov lus nyob rau nruab nrab kom muaj ntau tshaj ib me ntsis ntawm ib cov lus me ntsis, tab sis ib qho nqi tsawg dua ib ntsis ntawm cov ntaub ntawv ib ntsis yuav ua tau tiav.. cov lus siv cov txheej txheem encoding tsim nyog. Lub entropy ntawm cov lus nyob rau hauv me me lub sij hawm nws ntev yog ntsuas ntawm ntau npaum li cas cov ntaub ntawv nws muaj.
