The Semantic Web initiative has enabled a common infrastructure in knowledge representation for sharing knowledge of ontologies and instances through languages such as Resource Description Framework (RDF) and the Web Ontology Language (OWL). In this paper we present a framework for combining the shallow levels of semantic description commonly used in information extraction with the deeper semantic structures available in such ontology representation languages. In this approach named entities are considered as instances of concepts in a defined taxonomy, coreference relations as identity pools containing named entities and coreferring expressions, and events as predicateargument frames. The framework is being implemented within the Open Ontology Forge software for annotating media, currently text and images, in Web pages and locally stored document collections. We discuss the knowledge framework, some features of the system, and a road map for future development.